[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260806 03:59:07.317373 30646 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.237.190:35895
I20260806 03:59:07.318765 30646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:59:07.319865 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:59:07.338732 30656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:07.339512 30658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:07.340416 30654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:07.341158 30646 server_base.cc:1034] running on GCE node
I20260806 03:59:08.567435 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:59:08.567668 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:59:08.567823 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988748567800 us; error 0 us; skew 500 ppm
I20260806 03:59:08.568665 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:59:08.577172 30646 webserver.cc:460] Webserver started at http://127.29.237.190:36883/ using document root <none> and password file <none>
I20260806 03:59:08.578034 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 03:59:08.578225 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:59:08.578619 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:59:08.584913 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/master-0-root/instance:
uuid: "b8e734a12cb14d29818b0f4f32a9beb0"
format_stamp: "Formatted at 2026-08-06 03:59:08 on dist-test-slave-lgr5"
I20260806 03:59:08.592788 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 03:59:08.598030 30668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:08.599085 30646 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260806 03:59:08.599385 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/master-0-root
uuid: "b8e734a12cb14d29818b0f4f32a9beb0"
format_stamp: "Formatted at 2026-08-06 03:59:08 on dist-test-slave-lgr5"
I20260806 03:59:08.599674 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:59:08.636471 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:59:08.637943 30646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:59:08.638373 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:59:08.721078 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.190:35895
I20260806 03:59:08.721410 30751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.190:35895 every 8 connection(s)
I20260806 03:59:08.729696 30752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:08.753696 30752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0: Bootstrap starting.
I20260806 03:59:08.759986 30752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:08.761962 30752 log.cc:824] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0: Log is configured to *not* fsync() on all Append() calls
I20260806 03:59:08.766662 30752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0: No bootstrap required, opened a new log
I20260806 03:59:08.790757 30752 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e734a12cb14d29818b0f4f32a9beb0" member_type: VOTER }
I20260806 03:59:08.791332 30752 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:08.791627 30752 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e734a12cb14d29818b0f4f32a9beb0, State: Initialized, Role: FOLLOWER
I20260806 03:59:08.792631 30752 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e734a12cb14d29818b0f4f32a9beb0" member_type: VOTER }
I20260806 03:59:08.793154 30752 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:59:08.793437 30752 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:59:08.793756 30752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:08.797745 30752 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e734a12cb14d29818b0f4f32a9beb0" member_type: VOTER }
I20260806 03:59:08.798454 30752 leader_election.cc:302] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8e734a12cb14d29818b0f4f32a9beb0; no voters:
I20260806 03:59:08.800277 30752 leader_election.cc:288] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:59:08.800652 30755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:08.803544 30755 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 1 LEADER]: Becoming Leader. State: Replica: b8e734a12cb14d29818b0f4f32a9beb0, State: Running, Role: LEADER
I20260806 03:59:08.804471 30755 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e734a12cb14d29818b0f4f32a9beb0" member_type: VOTER }
I20260806 03:59:08.805866 30752 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:59:08.815735 30757 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8e734a12cb14d29818b0f4f32a9beb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e734a12cb14d29818b0f4f32a9beb0" member_type: VOTER } }
I20260806 03:59:08.815789 30758 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8e734a12cb14d29818b0f4f32a9beb0. Latest consensus state: current_term: 1 leader_uuid: "b8e734a12cb14d29818b0f4f32a9beb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e734a12cb14d29818b0f4f32a9beb0" member_type: VOTER } }
I20260806 03:59:08.816677 30757 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [sys.catalog]: This master's current role is: LEADER
I20260806 03:59:08.816677 30758 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [sys.catalog]: This master's current role is: LEADER
I20260806 03:59:08.820514 30765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:59:08.832145 30765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:59:08.838107 30646 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 03:59:08.851424 30765 catalog_manager.cc:1347] Generated new cluster ID: 1f7e6c9d995a4057aebc92a1b3b5270b
I20260806 03:59:08.851748 30765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:59:08.865269 30765 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:59:08.867357 30765 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:59:08.895876 30765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0: Generated new TSK 0
I20260806 03:59:08.896960 30765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:59:08.909754 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:59:08.919459 30782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:08.920917 30783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:08.924973 30785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:08.925117 30646 server_base.cc:1034] running on GCE node
I20260806 03:59:08.926057 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:59:08.926239 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:59:08.926409 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988748926388 us; error 0 us; skew 500 ppm
I20260806 03:59:08.926980 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:59:08.930998 30646 webserver.cc:460] Webserver started at http://127.29.237.129:37243/ using document root <none> and password file <none>
I20260806 03:59:08.931522 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 03:59:08.931715 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:59:08.931977 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:59:08.933343 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-0-root/instance:
uuid: "bde9de7562e24ecdb40f72bf3f753e15"
format_stamp: "Formatted at 2026-08-06 03:59:08 on dist-test-slave-lgr5"
I20260806 03:59:08.939689 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 03:59:08.943984 30791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:08.944871 30646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:59:08.945142 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-0-root
uuid: "bde9de7562e24ecdb40f72bf3f753e15"
format_stamp: "Formatted at 2026-08-06 03:59:08 on dist-test-slave-lgr5"
I20260806 03:59:08.945434 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:59:08.968129 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:59:08.969619 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:59:08.972453 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:59:08.977237 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:59:08.977474 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:08.977761 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:59:08.977926 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:09.082079 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.129:40233
I20260806 03:59:09.082192 30883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.129:40233 every 8 connection(s)
I20260806 03:59:09.092618 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:59:09.108879 30888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:09.109803 30889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:09.118525 30893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:09.122272 30646 server_base.cc:1034] running on GCE node
I20260806 03:59:09.123203 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:59:09.123418 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:59:09.123616 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988749123599 us; error 0 us; skew 500 ppm
I20260806 03:59:09.124256 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:59:09.128751 30884 heartbeater.cc:347] Connected to a master server at 127.29.237.190:35895
I20260806 03:59:09.129310 30884 heartbeater.cc:464] Registering TS with master...
I20260806 03:59:09.130563 30884 heartbeater.cc:511] Master 127.29.237.190:35895 requested a full tablet report, sending...
I20260806 03:59:09.133765 30646 webserver.cc:460] Webserver started at http://127.29.237.130:43697/ using document root <none> and password file <none>
I20260806 03:59:09.134488 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 03:59:09.134447 30698 ts_manager.cc:194] Registered new tserver with Master: bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233)
I20260806 03:59:09.134773 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:59:09.135138 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:59:09.136680 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-1-root/instance:
uuid: "098e0e271e91425cafa13cdd4f19f8b3"
format_stamp: "Formatted at 2026-08-06 03:59:09 on dist-test-slave-lgr5"
I20260806 03:59:09.137724 30698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46576
I20260806 03:59:09.143440 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 03:59:09.148196 30901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:09.149056 30646 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 03:59:09.149323 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-1-root
uuid: "098e0e271e91425cafa13cdd4f19f8b3"
format_stamp: "Formatted at 2026-08-06 03:59:09 on dist-test-slave-lgr5"
I20260806 03:59:09.149643 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:59:09.184425 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:59:09.185698 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:59:09.187858 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:59:09.191078 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:59:09.191272 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:09.191501 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:59:09.191645 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:09.286996 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.130:34147
I20260806 03:59:09.287230 30983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.130:34147 every 8 connection(s)
I20260806 03:59:09.295428 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:59:09.303411 30990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:09.309583 30984 heartbeater.cc:347] Connected to a master server at 127.29.237.190:35895
I20260806 03:59:09.310038 30984 heartbeater.cc:464] Registering TS with master...
I20260806 03:59:09.311124 30984 heartbeater.cc:511] Master 127.29.237.190:35895 requested a full tablet report, sending...
W20260806 03:59:09.313213 30991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:09.314235 30698 ts_manager.cc:194] Registered new tserver with Master: 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:09.316586 30698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46582
W20260806 03:59:09.319808 30993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:09.328786 30646 server_base.cc:1034] running on GCE node
I20260806 03:59:09.329710 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:59:09.329905 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:59:09.330062 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988749330045 us; error 0 us; skew 500 ppm
I20260806 03:59:09.330533 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:59:09.333429 30646 webserver.cc:460] Webserver started at http://127.29.237.131:43477/ using document root <none> and password file <none>
I20260806 03:59:09.333913 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 03:59:09.334100 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:59:09.334334 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:59:09.335462 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-2-root/instance:
uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
format_stamp: "Formatted at 2026-08-06 03:59:09 on dist-test-slave-lgr5"
I20260806 03:59:09.339860 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 03:59:09.343115 31000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:09.343853 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 03:59:09.344131 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-2-root
uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
format_stamp: "Formatted at 2026-08-06 03:59:09 on dist-test-slave-lgr5"
I20260806 03:59:09.344390 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988747097609-30646-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:59:09.365638 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:59:09.366875 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:59:09.368666 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:59:09.371941 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:59:09.372152 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 03:59:09.372385 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:59:09.372581 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:09.467078 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.131:39679
I20260806 03:59:09.467705 31104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.131:39679 every 8 connection(s)
I20260806 03:59:09.486063 31106 heartbeater.cc:347] Connected to a master server at 127.29.237.190:35895
I20260806 03:59:09.486431 31106 heartbeater.cc:464] Registering TS with master...
I20260806 03:59:09.487191 31106 heartbeater.cc:511] Master 127.29.237.190:35895 requested a full tablet report, sending...
I20260806 03:59:09.489207 30698 ts_manager.cc:194] Registered new tserver with Master: f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:09.489316 30646 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015647638s
I20260806 03:59:09.491338 30698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46598
I20260806 03:59:09.492838 30646 test_util.cc:268] Using random seed: -386104751
I20260806 03:59:09.547820 30698 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46612:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:59:09.551692 30698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:59:09.704509 30938 tablet_service.cc:1467] Processing CreateTablet for tablet d0a734a4aa10450a905419d5968af802 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:59:09.704391 30940 tablet_service.cc:1467] Processing CreateTablet for tablet 5486ddbb7fc34b0da6220b44d27f0ecc (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:59:09.705907 30942 tablet_service.cc:1467] Processing CreateTablet for tablet 49a622c056db4064ba7658fc3db5315d (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:59:09.715984 30940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5486ddbb7fc34b0da6220b44d27f0ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.716302 30936 tablet_service.cc:1467] Processing CreateTablet for tablet 63c3ef8772aa477aadf447386445353b (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:59:09.714691 30934 tablet_service.cc:1467] Processing CreateTablet for tablet 3657570ad79d43c6a8dce96803512091 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:59:09.707942 30941 tablet_service.cc:1467] Processing CreateTablet for tablet b0eb5e025e0c43cf879ba801e672a0f1 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:59:09.723047 30938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0a734a4aa10450a905419d5968af802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.723579 30932 tablet_service.cc:1467] Processing CreateTablet for tablet 33400b2f7d8f4103ac04c30fa55fb96d (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:59:09.726691 30931 tablet_service.cc:1467] Processing CreateTablet for tablet 5d757fe41bf345f8a9ba96f6e4160dd1 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:59:09.720125 30939 tablet_service.cc:1467] Processing CreateTablet for tablet 14f89048e201458ebc05796417c0ae31 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:59:09.731491 30936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c3ef8772aa477aadf447386445353b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.741384 30931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d757fe41bf345f8a9ba96f6e4160dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.750257 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 49a622c056db4064ba7658fc3db5315d (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:59:09.753173 30942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a622c056db4064ba7658fc3db5315d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.753259 31051 tablet_service.cc:1467] Processing CreateTablet for tablet b0eb5e025e0c43cf879ba801e672a0f1 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:59:09.765319 30934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3657570ad79d43c6a8dce96803512091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.769745 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a622c056db4064ba7658fc3db5315d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.776124 30941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb5e025e0c43cf879ba801e672a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.762833 31049 tablet_service.cc:1467] Processing CreateTablet for tablet 14f89048e201458ebc05796417c0ae31 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:59:09.768553 31042 tablet_service.cc:1467] Processing CreateTablet for tablet 3657570ad79d43c6a8dce96803512091 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:59:09.785760 30939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f89048e201458ebc05796417c0ae31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.778736 30832 tablet_service.cc:1467] Processing CreateTablet for tablet 49a622c056db4064ba7658fc3db5315d (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:59:09.768524 31046 tablet_service.cc:1467] Processing CreateTablet for tablet d0a734a4aa10450a905419d5968af802 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:59:09.782150 30831 tablet_service.cc:1467] Processing CreateTablet for tablet b0eb5e025e0c43cf879ba801e672a0f1 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:59:09.784339 31041 tablet_service.cc:1467] Processing CreateTablet for tablet 33400b2f7d8f4103ac04c30fa55fb96d (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:59:09.796036 30932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33400b2f7d8f4103ac04c30fa55fb96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.779079 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 5486ddbb7fc34b0da6220b44d27f0ecc (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:59:09.796545 30826 tablet_service.cc:1467] Processing CreateTablet for tablet 14f89048e201458ebc05796417c0ae31 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:59:09.791009 30825 tablet_service.cc:1467] Processing CreateTablet for tablet d0a734a4aa10450a905419d5968af802 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:59:09.801163 30832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a622c056db4064ba7658fc3db5315d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.797729 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb5e025e0c43cf879ba801e672a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.786185 31040 tablet_service.cc:1467] Processing CreateTablet for tablet 5d757fe41bf345f8a9ba96f6e4160dd1 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:59:09.787756 31045 tablet_service.cc:1467] Processing CreateTablet for tablet 63c3ef8772aa477aadf447386445353b (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:59:09.799269 30823 tablet_service.cc:1467] Processing CreateTablet for tablet 63c3ef8772aa477aadf447386445353b (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:59:09.816859 30831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb5e025e0c43cf879ba801e672a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.798398 30827 tablet_service.cc:1467] Processing CreateTablet for tablet 5486ddbb7fc34b0da6220b44d27f0ecc (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:59:09.813422 30819 tablet_service.cc:1467] Processing CreateTablet for tablet 3657570ad79d43c6a8dce96803512091 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:59:09.819891 31042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3657570ad79d43c6a8dce96803512091. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:59:09.834779 30769 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:59:09.836026 30825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0a734a4aa10450a905419d5968af802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.837782 30817 tablet_service.cc:1467] Processing CreateTablet for tablet 5d757fe41bf345f8a9ba96f6e4160dd1 (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:59:09.859872 30827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5486ddbb7fc34b0da6220b44d27f0ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.863683 30823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c3ef8772aa477aadf447386445353b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.817957 30818 tablet_service.cc:1467] Processing CreateTablet for tablet 33400b2f7d8f4103ac04c30fa55fb96d (DEFAULT_TABLE table=test-workload [id=2216ff08793d41a395f34d1e0cf05423]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:59:09.873044 31045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c3ef8772aa477aadf447386445353b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.874778 30826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f89048e201458ebc05796417c0ae31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.882733 31040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d757fe41bf345f8a9ba96f6e4160dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.885044 31046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0a734a4aa10450a905419d5968af802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.891258 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5486ddbb7fc34b0da6220b44d27f0ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.894479 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f89048e201458ebc05796417c0ae31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.899843 30819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3657570ad79d43c6a8dce96803512091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.904271 30817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d757fe41bf345f8a9ba96f6e4160dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.908248 30818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33400b2f7d8f4103ac04c30fa55fb96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:09.904322 31041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33400b2f7d8f4103ac04c30fa55fb96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:10.142657 30884 heartbeater.cc:503] Master 127.29.237.190:35895 was elected leader, sending a full tablet report...
I20260806 03:59:10.174523 31138 tablet_bootstrap.cc:492] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:10.175408 31137 tablet_bootstrap.cc:492] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:10.184000 31139 tablet_bootstrap.cc:492] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:10.220808 31137 tablet_bootstrap.cc:654] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.222990 31139 tablet_bootstrap.cc:654] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.223559 31138 tablet_bootstrap.cc:654] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.241637 31137 tablet_bootstrap.cc:492] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:10.241751 31138 tablet_bootstrap.cc:492] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:10.242319 31137 ts_tablet_manager.cc:1397] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.101s user 0.018s sys 0.013s
I20260806 03:59:10.242475 31138 ts_tablet_manager.cc:1397] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.070s user 0.013s sys 0.014s
I20260806 03:59:10.244417 31139 tablet_bootstrap.cc:492] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:10.245069 31139 ts_tablet_manager.cc:1397] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.024s
I20260806 03:59:10.247013 31137 raft_consensus.cc:348] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.247452 31138 raft_consensus.cc:348] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.248070 31137 raft_consensus.cc:374] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.248409 31137 raft_consensus.cc:729] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.248409 31138 raft_consensus.cc:374] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.249426 31138 raft_consensus.cc:729] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.249018 31139 raft_consensus.cc:348] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.249234 31137 consensus_queue.cc:260] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.249903 31139 raft_consensus.cc:374] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.250231 31139 raft_consensus.cc:729] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.250316 31138 consensus_queue.cc:260] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.252260 31139 consensus_queue.cc:260] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.257700 31138 ts_tablet_manager.cc:1428] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.015s user 0.004s sys 0.006s
I20260806 03:59:10.258942 31138 tablet_bootstrap.cc:492] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:10.262427 31139 ts_tablet_manager.cc:1428] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.017s user 0.008s sys 0.003s
I20260806 03:59:10.262508 31137 ts_tablet_manager.cc:1428] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.020s user 0.005s sys 0.012s
I20260806 03:59:10.263259 31106 heartbeater.cc:503] Master 127.29.237.190:35895 was elected leader, sending a full tablet report...
I20260806 03:59:10.263621 31137 tablet_bootstrap.cc:492] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:10.267969 30984 heartbeater.cc:503] Master 127.29.237.190:35895 was elected leader, sending a full tablet report...
I20260806 03:59:10.268198 31139 tablet_bootstrap.cc:492] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:10.276292 31138 tablet_bootstrap.cc:654] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.283607 31137 tablet_bootstrap.cc:654] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.295351 31139 tablet_bootstrap.cc:654] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
W20260806 03:59:10.297804 30988 tablet.cc:2369] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:59:10.319775 31138 tablet_bootstrap.cc:492] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:10.320523 31138 ts_tablet_manager.cc:1397] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.035s
I20260806 03:59:10.321555 31139 tablet_bootstrap.cc:492] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:10.322247 31139 ts_tablet_manager.cc:1397] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.054s user 0.022s sys 0.016s
I20260806 03:59:10.323037 31137 tablet_bootstrap.cc:492] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:10.323607 31137 ts_tablet_manager.cc:1397] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.012s
I20260806 03:59:10.325136 31138 raft_consensus.cc:348] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.326144 31138 raft_consensus.cc:374] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.326557 31138 raft_consensus.cc:729] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.326745 31139 raft_consensus.cc:348] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.327982 31139 raft_consensus.cc:374] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.327786 31137 raft_consensus.cc:348] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.328920 31137 raft_consensus.cc:374] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.329237 31137 raft_consensus.cc:729] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.328679 31138 consensus_queue.cc:260] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.328480 31139 raft_consensus.cc:729] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.330818 31139 consensus_queue.cc:260] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.332104 31138 ts_tablet_manager.cc:1428] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 03:59:10.333231 31138 tablet_bootstrap.cc:492] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:10.333959 31139 ts_tablet_manager.cc:1428] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.011s user 0.002s sys 0.005s
I20260806 03:59:10.334959 31139 tablet_bootstrap.cc:492] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:10.343807 31137 consensus_queue.cc:260] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.346979 31137 ts_tablet_manager.cc:1428] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.023s user 0.007s sys 0.003s
I20260806 03:59:10.348173 31137 tablet_bootstrap.cc:492] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:10.353266 31138 tablet_bootstrap.cc:654] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.362291 31139 tablet_bootstrap.cc:654] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.367102 31137 tablet_bootstrap.cc:654] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.375367 31138 tablet_bootstrap.cc:492] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:10.376116 31138 ts_tablet_manager.cc:1397] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.020s
I20260806 03:59:10.381121 31138 raft_consensus.cc:348] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.382041 31138 raft_consensus.cc:374] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.382513 31138 raft_consensus.cc:729] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.382874 31139 tablet_bootstrap.cc:492] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:10.383550 31139 ts_tablet_manager.cc:1397] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.007s
I20260806 03:59:10.383590 31138 consensus_queue.cc:260] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.388633 31139 raft_consensus.cc:348] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.392479 31139 raft_consensus.cc:374] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.392899 31139 raft_consensus.cc:729] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.393867 31139 consensus_queue.cc:260] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.395864 31138 ts_tablet_manager.cc:1428] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.019s user 0.012s sys 0.000s
I20260806 03:59:10.397225 31138 tablet_bootstrap.cc:492] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:10.398466 31137 tablet_bootstrap.cc:492] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:10.398973 31137 ts_tablet_manager.cc:1397] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.020s
I20260806 03:59:10.403153 31139 ts_tablet_manager.cc:1428] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.019s user 0.009s sys 0.005s
I20260806 03:59:10.404611 31139 tablet_bootstrap.cc:492] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:10.403586 31137 raft_consensus.cc:348] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.408536 31137 raft_consensus.cc:374] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.408981 31137 raft_consensus.cc:729] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.409343 31138 tablet_bootstrap.cc:654] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.410115 31137 consensus_queue.cc:260] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.410894 31147 raft_consensus.cc:482] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.411556 31147 raft_consensus.cc:504] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.416364 31137 ts_tablet_manager.cc:1428] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.017s user 0.008s sys 0.005s
I20260806 03:59:10.418442 31147 leader_election.cc:288] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.424450 31137 tablet_bootstrap.cc:492] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:10.463899 31138 tablet_bootstrap.cc:492] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:10.465370 31139 tablet_bootstrap.cc:654] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.466619 31138 ts_tablet_manager.cc:1397] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.070s user 0.018s sys 0.015s
I20260806 03:59:10.471060 31137 tablet_bootstrap.cc:654] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.471275 31138 raft_consensus.cc:348] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.475024 31138 raft_consensus.cc:374] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.475198 31146 raft_consensus.cc:482] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.475492 31138 raft_consensus.cc:729] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.475848 31146 raft_consensus.cc:504] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.480165 31138 consensus_queue.cc:260] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.483098 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c3ef8772aa477aadf447386445353b" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:10.484952 31146 leader_election.cc:288] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
W20260806 03:59:10.492662 30794 leader_election.cc:341] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.500876 31138 ts_tablet_manager.cc:1428] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.034s user 0.009s sys 0.002s
I20260806 03:59:10.503659 31139 tablet_bootstrap.cc:492] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:10.504344 31139 ts_tablet_manager.cc:1397] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.100s user 0.014s sys 0.016s
I20260806 03:59:10.520939 31138 tablet_bootstrap.cc:492] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:10.542627 31139 raft_consensus.cc:348] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.541689 31147 raft_consensus.cc:482] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.543836 31139 raft_consensus.cc:374] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.544255 31139 raft_consensus.cc:729] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.543998 31147 raft_consensus.cc:504] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.545390 31139 consensus_queue.cc:260] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.549402 31139 ts_tablet_manager.cc:1428] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.045s user 0.005s sys 0.008s
I20260806 03:59:10.550946 31139 tablet_bootstrap.cc:492] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:10.546913 31147 leader_election.cc:288] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.556223 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f89048e201458ebc05796417c0ae31" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
W20260806 03:59:10.560389 30794 leader_election.cc:341] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.577754 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c3ef8772aa477aadf447386445353b" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.579577 31149 raft_consensus.cc:482] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.580641 31149 raft_consensus.cc:504] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
W20260806 03:59:10.583572 30795 leader_election.cc:341] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.584131 31146 raft_consensus.cc:482] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.584074 30795 leader_election.cc:302] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15; no voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:10.587064 31147 raft_consensus.cc:2726] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:10.584802 31146 raft_consensus.cc:504] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.590824 31146 leader_election.cc:288] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.592872 31138 tablet_bootstrap.cc:654] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.591778 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f89048e201458ebc05796417c0ae31" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.594992 31149 leader_election.cc:288] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:10.593756 31067 raft_consensus.cc:2454] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 0.
I20260806 03:59:10.613294 31139 tablet_bootstrap.cc:654] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.621518 31137 tablet_bootstrap.cc:492] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:10.624284 31137 ts_tablet_manager.cc:1397] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.206s user 0.017s sys 0.022s
I20260806 03:59:10.639842 30795 leader_election.cc:302] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters: 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:10.642526 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.643661 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d757fe41bf345f8a9ba96f6e4160dd1" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.644909 31147 raft_consensus.cc:2781] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:10.640473 31137 raft_consensus.cc:348] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.646350 31137 raft_consensus.cc:374] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 03:59:10.645934 30906 leader_election.cc:341] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.645327 31147 raft_consensus.cc:482] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:10.648970 31147 raft_consensus.cc:3037] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:10.648353 31067 raft_consensus.cc:2454] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 0.
I20260806 03:59:10.646735 31137 raft_consensus.cc:729] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.651844 30906 leader_election.cc:302] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:10.654697 31146 raft_consensus.cc:2781] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:10.655117 31146 raft_consensus.cc:482] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:10.657059 31149 raft_consensus.cc:482] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.657878 31146 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:10.657550 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:10.657656 31149 raft_consensus.cc:504] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.662371 31149 leader_election.cc:288] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:10.656309 31137 consensus_queue.cc:260] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.668144 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d757fe41bf345f8a9ba96f6e4160dd1" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:10.669943 31138 tablet_bootstrap.cc:492] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:10.670547 31138 ts_tablet_manager.cc:1397] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.150s user 0.010s sys 0.018s
I20260806 03:59:10.675208 31146 raft_consensus.cc:504] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
W20260806 03:59:10.674224 30907 leader_election.cc:341] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.678901 31147 raft_consensus.cc:504] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
W20260806 03:59:10.679970 30907 leader_election.cc:341] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.680665 30907 leader_election.cc:302] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3; no voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:10.685696 31147 leader_election.cc:288] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.689539 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f89048e201458ebc05796417c0ae31" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:10.690272 31067 raft_consensus.cc:3037] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:10.690478 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:10.697472 31138 raft_consensus.cc:348] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:10.697912 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:10.698311 31138 raft_consensus.cc:374] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.698520 31068 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:10.698670 31138 raft_consensus.cc:729] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.691958 31146 leader_election.cc:288] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.705003 31146 raft_consensus.cc:482] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.705440 31139 tablet_bootstrap.cc:492] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:10.706110 31139 ts_tablet_manager.cc:1397] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.156s user 0.024s sys 0.023s
I20260806 03:59:10.705689 31146 raft_consensus.cc:504] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.710429 31146 leader_election.cc:288] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.711969 31066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.714520 31139 raft_consensus.cc:348] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.701309 31138 consensus_queue.cc:260] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:10.717012 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:10.718840 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f89048e201458ebc05796417c0ae31" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
W20260806 03:59:10.716142 30906 leader_election.cc:341] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.717134 31139 raft_consensus.cc:374] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 03:59:10.720340 30794 leader_election.cc:341] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.720590 31139 raft_consensus.cc:729] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.721619 31139 consensus_queue.cc:260] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.724610 31138 ts_tablet_manager.cc:1428] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.054s user 0.012s sys 0.006s
I20260806 03:59:10.713994 31146 raft_consensus.cc:2726] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:59:10.727260 30907 leader_election.cc:341] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.811175 31067 raft_consensus.cc:2454] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 1.
W20260806 03:59:10.811405 30907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:40233, user_credentials={real_user=slave}} blocked reactor thread for 84515.2us
I20260806 03:59:10.810114 31137 ts_tablet_manager.cc:1428] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.183s user 0.032s sys 0.039s
I20260806 03:59:10.725781 31139 ts_tablet_manager.cc:1428] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.019s user 0.007s sys 0.004s
I20260806 03:59:10.788750 31185 raft_consensus.cc:482] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.756593 31147 raft_consensus.cc:482] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:59:10.813597 30907 leader_election.cc:341] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.821882 31138 tablet_bootstrap.cc:492] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:10.821446 31137 tablet_bootstrap.cc:492] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:10.823781 31139 tablet_bootstrap.cc:492] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:10.826148 30907 leader_election.cc:302] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3; no voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:10.824520 31149 raft_consensus.cc:482] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.828327 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:10.829353 31146 raft_consensus.cc:2726] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:10.828209 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f89048e201458ebc05796417c0ae31" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:10.823007 31147 raft_consensus.cc:504] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:10.829234 31149 raft_consensus.cc:504] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.835664 31149 leader_election.cc:288] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:10.832038 30795 leader_election.cc:302] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters: 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:10.827382 31068 raft_consensus.cc:2454] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 1.
I20260806 03:59:10.817648 31185 raft_consensus.cc:504] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.837574 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f89048e201458ebc05796417c0ae31" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:10.840601 30951 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 0.
I20260806 03:59:10.840989 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:10.840982 30906 leader_election.cc:302] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters: bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:10.844508 30844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:10.847349 30846 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 0.
W20260806 03:59:10.849383 31007 leader_election.cc:341] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.850862 30847 raft_consensus.cc:2379] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in current term 1: Already voted for candidate bde9de7562e24ecdb40f72bf3f753e15 in this term.
I20260806 03:59:10.848611 31146 raft_consensus.cc:2781] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 03:59:10.854611 30769 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:59:10.857520 31186 raft_consensus.cc:482] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.853269 31146 raft_consensus.cc:686] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:10.858286 31186 raft_consensus.cc:504] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.868496 31147 leader_election.cc:288] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:10.869122 31147 raft_consensus.cc:482] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.869930 31147 raft_consensus.cc:504] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
W20260806 03:59:10.867911 31003 leader_election.cc:341] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.869278 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:10.873239 31146 consensus_queue.cc:237] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.873931 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:10.874545 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
W20260806 03:59:10.873939 30794 leader_election.cc:341] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:59:10.876595 30794 leader_election.cc:341] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.875208 31185 leader_election.cc:288] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.876745 31003 leader_election.cc:302] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:10.877295 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.878340 31149 raft_consensus.cc:2726] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:10.879040 31138 tablet_bootstrap.cc:654] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.878352 31068 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 0.
I20260806 03:59:10.879819 31007 leader_election.cc:302] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:10.880894 30906 leader_election.cc:302] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:10.880780 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.882256 31185 raft_consensus.cc:2781] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:10.882613 31185 raft_consensus.cc:482] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:10.882396 30846 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 0.
I20260806 03:59:10.883140 31185 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:10.883844 31147 leader_election.cc:288] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.876259 31187 raft_consensus.cc:2781] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:10.882105 31137 tablet_bootstrap.cc:654] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.885677 31149 raft_consensus.cc:2781] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:10.886039 31149 raft_consensus.cc:482] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:10.886407 31149 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:10.884963 31187 raft_consensus.cc:686] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Becoming Leader. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:10.889199 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.889725 31139 tablet_bootstrap.cc:654] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
W20260806 03:59:10.890431 30795 leader_election.cc:341] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:59:10.881385 31003 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:34147, user_credentials={real_user=slave}} blocked reactor thread for 47393.1us
I20260806 03:59:10.891207 30795 leader_election.cc:302] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15; no voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b
W20260806 03:59:10.894536 30795 leader_election.cc:341] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.895871 31147 raft_consensus.cc:2726] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:10.895193 30795 leader_election.cc:302] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15; no voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:10.898479 31147 raft_consensus.cc:2726] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:10.902232 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:10.902786 31186 leader_election.cc:288] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.890678 31187 consensus_queue.cc:237] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.905087 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:10.906108 31149 raft_consensus.cc:504] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.907922 31185 raft_consensus.cc:504] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.908952 30698 catalog_manager.cc:5495] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:10.910297 31149 leader_election.cc:288] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:10.910497 30951 raft_consensus.cc:2379] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bde9de7562e24ecdb40f72bf3f753e15 in current term 1: Already voted for candidate 098e0e271e91425cafa13cdd4f19f8b3 in this term.
W20260806 03:59:10.916499 31003 leader_election.cc:341] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.917142 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:10.917614 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.917892 30846 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:10.919961 31067 raft_consensus.cc:2379] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bde9de7562e24ecdb40f72bf3f753e15 in current term 1: Already voted for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in this term.
I20260806 03:59:10.921602 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:10.918586 31003 leader_election.cc:302] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:10.925235 30795 leader_election.cc:302] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15; no voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:10.925698 31149 raft_consensus.cc:2726] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:10.929260 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:10.930653 30951 raft_consensus.cc:2379] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in current term 1: Already voted for candidate 098e0e271e91425cafa13cdd4f19f8b3 in this term.
I20260806 03:59:10.937786 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:10.938913 31067 raft_consensus.cc:2379] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 098e0e271e91425cafa13cdd4f19f8b3 in current term 1: Already voted for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in this term.
I20260806 03:59:10.938334 31185 leader_election.cc:288] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.942955 30906 leader_election.cc:302] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3; no voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:10.944578 31185 raft_consensus.cc:2726] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:10.952221 30846 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 1.
I20260806 03:59:10.953184 31186 raft_consensus.cc:2726] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:10.954650 31007 leader_election.cc:302] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters: 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:10.955785 31149 raft_consensus.cc:2781] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:10.957803 31138 tablet_bootstrap.cc:492] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:10.957284 31137 tablet_bootstrap.cc:492] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:10.958452 31138 ts_tablet_manager.cc:1397] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.144s user 0.008s sys 0.015s
I20260806 03:59:10.961746 31149 raft_consensus.cc:686] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Becoming Leader. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:10.962558 31137 ts_tablet_manager.cc:1397] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.147s user 0.005s sys 0.019s
I20260806 03:59:10.958276 30695 catalog_manager.cc:5495] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 reported cstate change: term changed from 0 to 1, leader changed from <none> to bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:10.962998 31149 consensus_queue.cc:237] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.966933 31139 tablet_bootstrap.cc:492] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:10.969807 31139 ts_tablet_manager.cc:1397] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.152s user 0.013s sys 0.029s
I20260806 03:59:10.972590 31139 raft_consensus.cc:348] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.975340 31139 raft_consensus.cc:374] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.975756 31139 raft_consensus.cc:729] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.968906 31138 raft_consensus.cc:348] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.976682 31138 raft_consensus.cc:374] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.977028 31138 raft_consensus.cc:729] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.977733 31138 consensus_queue.cc:260] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.974509 31137 raft_consensus.cc:348] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.980322 31137 raft_consensus.cc:374] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:10.980733 31137 raft_consensus.cc:729] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:10.981968 31186 raft_consensus.cc:482] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:10.982453 31186 raft_consensus.cc:504] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.984431 31186 leader_election.cc:288] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:10.985445 31138 ts_tablet_manager.cc:1428] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.023s user 0.003s sys 0.003s
I20260806 03:59:10.987934 31138 tablet_bootstrap.cc:492] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:10.986830 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:10.989105 31067 raft_consensus.cc:2454] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 0.
I20260806 03:59:10.991063 30795 leader_election.cc:302] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:10.991967 31186 raft_consensus.cc:2781] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:10.992563 31186 raft_consensus.cc:482] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:10.992905 31186 raft_consensus.cc:3037] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:10.995501 31138 tablet_bootstrap.cc:654] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:10.985697 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:10.985949 31137 consensus_queue.cc:260] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
W20260806 03:59:10.997898 30794 leader_election.cc:341] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:10.998436 31137 ts_tablet_manager.cc:1428] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.035s user 0.004s sys 0.002s
I20260806 03:59:10.999433 31137 tablet_bootstrap.cc:492] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:11.001240 31186 raft_consensus.cc:504] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:10.981312 31139 consensus_queue.cc:260] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.004371 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
W20260806 03:59:11.005811 30794 leader_election.cc:341] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:11.005869 31139 ts_tablet_manager.cc:1428] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.036s user 0.004s sys 0.001s
I20260806 03:59:11.007220 31139 tablet_bootstrap.cc:492] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:11.012455 31137 tablet_bootstrap.cc:654] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.014101 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:11.013666 31186 leader_election.cc:288] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:11.014914 31067 raft_consensus.cc:3037] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.015278 30694 catalog_manager.cc:5495] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:11.020774 31139 tablet_bootstrap.cc:654] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.022416 31067 raft_consensus.cc:2454] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 1.
I20260806 03:59:11.023543 30795 leader_election.cc:302] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters: 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:11.024863 31186 raft_consensus.cc:2781] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:11.025624 31186 raft_consensus.cc:686] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Becoming Leader. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:11.027053 31186 consensus_queue.cc:237] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.030128 31149 raft_consensus.cc:482] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:11.030563 31149 raft_consensus.cc:504] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.032446 31149 leader_election.cc:288] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:11.034296 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:11.034937 30951 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 0.
I20260806 03:59:11.036278 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:11.036369 31003 leader_election.cc:302] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:11.037402 31149 raft_consensus.cc:2781] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 03:59:11.037649 31007 leader_election.cc:341] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:11.037779 31149 raft_consensus.cc:482] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:11.038190 31149 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.041532 31137 tablet_bootstrap.cc:492] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:11.043210 31137 ts_tablet_manager.cc:1397] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.000s
I20260806 03:59:11.046919 31138 tablet_bootstrap.cc:492] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:11.045863 31149 raft_consensus.cc:504] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.047739 31139 tablet_bootstrap.cc:492] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:11.048251 31139 ts_tablet_manager.cc:1397] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.007s
I20260806 03:59:11.047382 31138 ts_tablet_manager.cc:1397] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.060s user 0.020s sys 0.004s
I20260806 03:59:11.050206 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:11.053050 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:11.053650 30951 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.056406 31139 raft_consensus.cc:348] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.057135 31139 raft_consensus.cc:374] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.057410 31139 raft_consensus.cc:729] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.058169 31139 consensus_queue.cc:260] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.060619 30951 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 1.
I20260806 03:59:11.060755 31139 ts_tablet_manager.cc:1428] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.012s user 0.002s sys 0.005s
I20260806 03:59:11.053062 31137 raft_consensus.cc:348] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.061764 31003 leader_election.cc:302] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:11.063038 31137 raft_consensus.cc:374] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.063505 31199 raft_consensus.cc:2781] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:11.063730 31137 raft_consensus.cc:729] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.064714 31149 leader_election.cc:288] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:11.066427 31138 raft_consensus.cc:348] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.069237 31138 raft_consensus.cc:374] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.069541 31138 raft_consensus.cc:729] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.070151 31138 consensus_queue.cc:260] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.070988 30846 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.071576 31137 consensus_queue.cc:260] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.062037 31139 tablet_bootstrap.cc:492] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:11.077860 30846 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 1.
I20260806 03:59:11.078321 30695 catalog_manager.cc:5495] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 reported cstate change: term changed from 0 to 1, leader changed from <none> to bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:11.082574 31137 ts_tablet_manager.cc:1428] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260806 03:59:11.083290 31138 ts_tablet_manager.cc:1428] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.033s user 0.002s sys 0.006s
I20260806 03:59:11.083526 31137 tablet_bootstrap.cc:492] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:11.085659 31138 tablet_bootstrap.cc:492] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:11.092401 31138 tablet_bootstrap.cc:654] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.094895 31137 tablet_bootstrap.cc:654] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.097743 31199 raft_consensus.cc:686] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Becoming Leader. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:11.105990 31138 tablet_bootstrap.cc:492] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:11.105763 31199 consensus_queue.cc:237] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.106438 31138 ts_tablet_manager.cc:1397] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.007s
I20260806 03:59:11.111517 31138 raft_consensus.cc:348] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.112202 31138 raft_consensus.cc:374] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.112453 31138 raft_consensus.cc:729] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.113163 31138 consensus_queue.cc:260] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.114259 31139 tablet_bootstrap.cc:654] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.115461 31138 ts_tablet_manager.cc:1428] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.009s user 0.000s sys 0.004s
I20260806 03:59:11.119414 31138 tablet_bootstrap.cc:492] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15: Bootstrap starting.
I20260806 03:59:11.123600 31137 tablet_bootstrap.cc:492] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:11.125329 31137 ts_tablet_manager.cc:1397] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.018s
I20260806 03:59:11.126174 31214 raft_consensus.cc:482] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:11.126721 31214 raft_consensus.cc:504] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.130057 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d757fe41bf345f8a9ba96f6e4160dd1" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:11.130250 31214 leader_election.cc:288] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:11.130460 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d757fe41bf345f8a9ba96f6e4160dd1" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:11.130749 30951 raft_consensus.cc:2454] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 0.
I20260806 03:59:11.131177 30846 raft_consensus.cc:2454] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 0.
I20260806 03:59:11.132233 31003 leader_election.cc:302] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:11.133493 31214 raft_consensus.cc:2781] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:11.133845 31214 raft_consensus.cc:482] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:11.134168 31214 raft_consensus.cc:3037] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.138877 31139 tablet_bootstrap.cc:492] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:11.139427 31139 ts_tablet_manager.cc:1397] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.017s
I20260806 03:59:11.140265 31138 tablet_bootstrap.cc:654] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.141184 31214 raft_consensus.cc:504] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.142263 31139 raft_consensus.cc:348] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.143002 31139 raft_consensus.cc:374] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.143385 31139 raft_consensus.cc:729] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.143170 30694 catalog_manager.cc:5495] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:11.143931 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d757fe41bf345f8a9ba96f6e4160dd1" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:11.144055 31139 consensus_queue.cc:260] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.144629 30846 raft_consensus.cc:3037] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.144474 31137 raft_consensus.cc:348] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.145247 31137 raft_consensus.cc:374] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.145537 31137 raft_consensus.cc:729] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.146580 31137 consensus_queue.cc:260] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.149967 31139 ts_tablet_manager.cc:1428] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260806 03:59:11.148903 31185 raft_consensus.cc:482] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:11.151614 31139 tablet_bootstrap.cc:492] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b: Bootstrap starting.
I20260806 03:59:11.151702 31185 raft_consensus.cc:504] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.155057 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d757fe41bf345f8a9ba96f6e4160dd1" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:11.155637 30951 raft_consensus.cc:3037] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.155958 31185 leader_election.cc:288] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:11.149987 31214 leader_election.cc:288] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:11.158051 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:11.158344 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:11.159047 31067 raft_consensus.cc:2379] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 098e0e271e91425cafa13cdd4f19f8b3 in current term 1: Already voted for candidate bde9de7562e24ecdb40f72bf3f753e15 in this term.
I20260806 03:59:11.160773 31137 ts_tablet_manager.cc:1428] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.035s user 0.006s sys 0.001s
I20260806 03:59:11.161732 31137 tablet_bootstrap.cc:492] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:11.162006 30907 leader_election.cc:302] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3; no voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:11.164565 31138 tablet_bootstrap.cc:492] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15: No bootstrap required, opened a new log
I20260806 03:59:11.164937 31185 raft_consensus.cc:3037] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.162082 30951 raft_consensus.cc:2454] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 1.
I20260806 03:59:11.165020 31138 ts_tablet_manager.cc:1397] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.006s
I20260806 03:59:11.169512 31003 leader_election.cc:302] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:11.168068 30846 raft_consensus.cc:2454] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 1.
I20260806 03:59:11.171227 31214 raft_consensus.cc:2781] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:11.171943 31214 raft_consensus.cc:686] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Becoming Leader. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:11.172775 31214 consensus_queue.cc:237] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.173743 31185 raft_consensus.cc:2726] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:11.175159 31139 tablet_bootstrap.cc:654] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.175702 31137 tablet_bootstrap.cc:654] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.181587 31138 raft_consensus.cc:348] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.182276 31138 raft_consensus.cc:374] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.182533 31138 raft_consensus.cc:729] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.181969 30698 catalog_manager.cc:5495] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:11.183122 31138 consensus_queue.cc:260] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.186018 31137 tablet_bootstrap.cc:492] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:11.186452 31137 ts_tablet_manager.cc:1397] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20260806 03:59:11.188985 31137 raft_consensus.cc:348] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.189611 31137 raft_consensus.cc:374] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.189846 31137 raft_consensus.cc:729] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.190392 31137 consensus_queue.cc:260] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.195775 31138 ts_tablet_manager.cc:1428] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15: Time spent starting tablet: real 0.027s user 0.001s sys 0.003s
I20260806 03:59:11.195789 31139 tablet_bootstrap.cc:492] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b: No bootstrap required, opened a new log
I20260806 03:59:11.196964 31137 ts_tablet_manager.cc:1428] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260806 03:59:11.201371 31137 tablet_bootstrap.cc:492] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3: Bootstrap starting.
I20260806 03:59:11.206393 31137 tablet_bootstrap.cc:654] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:11.200506 31139 ts_tablet_manager.cc:1397] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.007s
I20260806 03:59:11.209981 31139 raft_consensus.cc:348] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.210680 31139 raft_consensus.cc:374] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.210948 31139 raft_consensus.cc:729] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.211164 31137 tablet_bootstrap.cc:492] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3: No bootstrap required, opened a new log
I20260806 03:59:11.211684 31137 ts_tablet_manager.cc:1397] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20260806 03:59:11.211545 31139 consensus_queue.cc:260] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.213605 31139 ts_tablet_manager.cc:1428] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 03:59:11.213647 31137 raft_consensus.cc:348] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.214243 31137 raft_consensus.cc:374] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:11.214454 31137 raft_consensus.cc:729] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Initialized, Role: FOLLOWER
I20260806 03:59:11.215479 31137 consensus_queue.cc:260] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.219391 31137 ts_tablet_manager.cc:1428] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 03:59:11.236529 31186 raft_consensus.cc:482] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:11.236945 31186 raft_consensus.cc:504] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.238399 31186 leader_election.cc:288] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:11.239334 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:11.239533 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:11.240370 31067 raft_consensus.cc:2379] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bde9de7562e24ecdb40f72bf3f753e15 in current term 1: Already voted for candidate 098e0e271e91425cafa13cdd4f19f8b3 in this term.
I20260806 03:59:11.241245 30795 leader_election.cc:302] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15; no voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:11.242025 31186 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.246496 31186 raft_consensus.cc:2726] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:11.322830 31185 consensus_queue.cc:1035] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.334712 31185 consensus_queue.cc:1035] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.355996 31185 raft_consensus.cc:482] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:11.356453 31185 raft_consensus.cc:504] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.358059 31185 leader_election.cc:288] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:11.358904 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:11.359540 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:11.361150 31067 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 0.
I20260806 03:59:11.359630 30846 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 0.
I20260806 03:59:11.362587 30907 leader_election.cc:302] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:11.363579 31185 raft_consensus.cc:2781] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:11.363842 31185 raft_consensus.cc:482] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:11.364100 31185 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.368392 31185 raft_consensus.cc:504] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.369859 31185 leader_election.cc:288] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:11.370599 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:11.371089 30846 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.370930 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:11.371552 31067 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.375198 30846 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 1.
I20260806 03:59:11.376035 30907 leader_election.cc:302] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:11.376677 31185 raft_consensus.cc:2781] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:11.377182 31185 raft_consensus.cc:686] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:11.377918 31067 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 1.
I20260806 03:59:11.378315 31185 consensus_queue.cc:237] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.383790 31214 raft_consensus.cc:482] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:11.384295 31214 raft_consensus.cc:504] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.385514 30694 catalog_manager.cc:5495] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:11.386384 31214 leader_election.cc:288] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:11.387115 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c3ef8772aa477aadf447386445353b" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:11.387723 30846 raft_consensus.cc:2454] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 0.
I20260806 03:59:11.388106 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c3ef8772aa477aadf447386445353b" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3" is_pre_election: true
I20260806 03:59:11.388686 31007 leader_election.cc:302] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:11.388695 30951 raft_consensus.cc:2454] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 0.
I20260806 03:59:11.389593 31214 raft_consensus.cc:2781] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:11.389926 31214 raft_consensus.cc:482] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:11.390223 31214 raft_consensus.cc:3037] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.394311 30951 raft_consensus.cc:3037] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.396548 31214 raft_consensus.cc:504] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.398344 31214 leader_election.cc:288] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:11.399055 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c3ef8772aa477aadf447386445353b" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:11.399438 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c3ef8772aa477aadf447386445353b" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:11.399613 30846 raft_consensus.cc:3037] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.400031 30952 raft_consensus.cc:3037] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.403437 31186 consensus_queue.cc:1035] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.405803 30846 raft_consensus.cc:2454] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 1.
I20260806 03:59:11.408231 31007 leader_election.cc:302] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:11.409053 31214 raft_consensus.cc:2781] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:11.409526 31214 raft_consensus.cc:686] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Becoming Leader. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:11.410364 31214 consensus_queue.cc:237] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.415702 31186 consensus_queue.cc:1035] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.418856 30952 raft_consensus.cc:2454] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 1.
I20260806 03:59:11.423468 31199 consensus_queue.cc:1035] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.426775 30698 catalog_manager.cc:5495] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:11.459333 31185 raft_consensus.cc:482] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:11.459815 31185 raft_consensus.cc:504] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.461854 31185 leader_election.cc:288] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679), bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233)
I20260806 03:59:11.463861 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15" is_pre_election: true
I20260806 03:59:11.465610 30846 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 0.
I20260806 03:59:11.466080 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" is_pre_election: true
I20260806 03:59:11.466116 31199 consensus_queue.cc:1035] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:59:11.466858 31067 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 0.
I20260806 03:59:11.466766 30907 leader_election.cc:302] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:11.467797 31185 raft_consensus.cc:2781] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:11.468501 31185 raft_consensus.cc:482] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:11.468876 31185 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.475380 31185 raft_consensus.cc:504] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.481354 31185 leader_election.cc:288] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Requested vote from peers f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679), bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233)
I20260806 03:59:11.481742 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:11.481953 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:11.482374 31067 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.482564 30846 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:11.489468 31067 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 1.
I20260806 03:59:11.489878 30846 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 1.
I20260806 03:59:11.490536 30906 leader_election.cc:302] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:11.491323 31185 raft_consensus.cc:2781] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:11.491693 31185 raft_consensus.cc:686] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:11.492419 31185 consensus_queue.cc:237] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.499686 30694 catalog_manager.cc:5495] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:11.510120 31214 consensus_queue.cc:1035] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.511435 31199 consensus_queue.cc:1035] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.514693 31187 consensus_queue.cc:1035] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.535791 31187 consensus_queue.cc:1035] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.583953 30950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
mode: GRACEFUL
new_leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:40318
I20260806 03:59:11.584578 30950 raft_consensus.cc:595] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Received request to transfer leadership to TS bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:11.600991 31067 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
mode: GRACEFUL
new_leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:34274
I20260806 03:59:11.601625 31067 raft_consensus.cc:595] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Received request to transfer leadership to TS bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:11.606424 30950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
mode: GRACEFUL
new_leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:40318
I20260806 03:59:11.607082 30950 raft_consensus.cc:595] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Received request to transfer leadership to TS bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:11.625484 30846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0a734a4aa10450a905419d5968af802"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
mode: GRACEFUL
new_leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:55080
I20260806 03:59:11.626006 30846 raft_consensus.cc:595] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Received request to transfer leadership to TS 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:11.630801 31067 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63c3ef8772aa477aadf447386445353b"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
mode: GRACEFUL
new_leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:34274
I20260806 03:59:11.631289 31067 raft_consensus.cc:595] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Received request to transfer leadership to TS 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:11.635798 31067 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3657570ad79d43c6a8dce96803512091"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
mode: GRACEFUL
new_leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:34274
I20260806 03:59:11.636330 31067 raft_consensus.cc:595] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Received request to transfer leadership to TS 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:11.664271 31214 consensus_queue.cc:1035] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.675830 31149 consensus_queue.cc:1035] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.809638 31185 consensus_queue.cc:1035] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.816334 31182 raft_consensus.cc:979] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3: : Instructing follower bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:11.816820 31182 raft_consensus.cc:1067] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Signalling peer bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:11.823932 31182 consensus_queue.cc:1035] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:59:11.825408 30846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:55052
I20260806 03:59:11.829769 30846 raft_consensus.cc:482] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:11.830258 30846 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.839650 30846 raft_consensus.cc:504] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.841845 30846 leader_election.cc:288] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 2 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:11.842957 30950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:11.843585 30950 raft_consensus.cc:3032] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:11.843988 30950 raft_consensus.cc:729] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:11.843791 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:11.844712 31067 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.848176 30950 consensus_queue.cc:260] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.849673 30950 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.856776 30950 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 2.
I20260806 03:59:11.857973 30794 leader_election.cc:302] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:11.858861 31241 raft_consensus.cc:2781] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:11.859433 31241 raft_consensus.cc:686] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Becoming Leader. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:11.860177 31067 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 2.
I20260806 03:59:11.861131 31149 consensus_queue.cc:1035] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.860538 31241 consensus_queue.cc:237] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.874270 30695 catalog_manager.cc:5495] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 reported cstate change: term changed from 1 to 2, leader changed from 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130) to bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:11.881093 31185 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.884262 31214 consensus_queue.cc:1035] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.903146 31146 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:11.913161 31214 raft_consensus.cc:979] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b: : Instructing follower 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:11.913787 31214 raft_consensus.cc:1067] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Signalling peer 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:11.916484 30950 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63c3ef8772aa477aadf447386445353b"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:40302
I20260806 03:59:11.917057 30950 raft_consensus.cc:482] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:11.917539 30950 raft_consensus.cc:3037] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.926745 30950 raft_consensus.cc:504] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.934715 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c3ef8772aa477aadf447386445353b" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:11.935480 30847 raft_consensus.cc:3037] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.937896 31146 raft_consensus.cc:979] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3: : Instructing follower bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:11.938378 31182 raft_consensus.cc:1067] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Signalling peer bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:11.941013 30846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:55052
I20260806 03:59:11.945339 30846 raft_consensus.cc:482] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:11.945796 30846 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.950058 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c3ef8772aa477aadf447386445353b" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:11.950975 31067 raft_consensus.cc:3032] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:11.951472 31067 raft_consensus.cc:729] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:11.951797 30847 raft_consensus.cc:2454] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 2.
I20260806 03:59:11.953833 30907 leader_election.cc:302] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:11.955924 30846 raft_consensus.cc:504] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.955924 31067 consensus_queue.cc:260] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.957870 31067 raft_consensus.cc:3037] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.958225 31182 raft_consensus.cc:2781] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:11.958922 31182 raft_consensus.cc:686] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:11.960006 30846 leader_election.cc:288] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 2 election: Requested vote from peers f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:11.960769 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:11.961014 30950 leader_election.cc:288] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 2 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:11.960132 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:11.965368 31068 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.961373 30952 raft_consensus.cc:3032] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:11.966336 30952 raft_consensus.cc:729] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:11.967206 30952 consensus_queue.cc:260] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.969123 31067 raft_consensus.cc:2454] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 2.
I20260806 03:59:11.969774 31199 raft_consensus.cc:979] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: : Instructing follower bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:11.970243 31199 raft_consensus.cc:1067] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Signalling peer bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:11.972201 30846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:55068
I20260806 03:59:11.972944 30846 raft_consensus.cc:482] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:11.972352 31182 consensus_queue.cc:237] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.973678 30846 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.979552 31068 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 2.
I20260806 03:59:11.969076 30952 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:11.982028 30795 leader_election.cc:302] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:11.986203 31241 raft_consensus.cc:2781] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:11.986783 31241 raft_consensus.cc:686] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Becoming Leader. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:11.987937 31241 consensus_queue.cc:237] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:11.989831 30695 catalog_manager.cc:5495] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 1 to 2, leader changed from f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131) to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:11.991291 30846 raft_consensus.cc:504] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:11.992067 30952 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 2.
I20260806 03:59:11.994963 30846 leader_election.cc:288] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 2 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:11.994982 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:11.998625 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:11.999297 31068 raft_consensus.cc:3032] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:11.999641 31068 raft_consensus.cc:729] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:11.999724 30951 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:12.000780 31068 consensus_queue.cc:260] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:12.002601 31068 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:12.011265 30951 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 2.
I20260806 03:59:12.011937 31068 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 2.
I20260806 03:59:12.013587 30795 leader_election.cc:302] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:12.015558 31241 raft_consensus.cc:2781] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:12.016583 31241 raft_consensus.cc:686] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Becoming Leader. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:12.018007 31241 consensus_queue.cc:237] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:12.019800 31149 raft_consensus.cc:979] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b: : Instructing follower 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:12.020499 31214 raft_consensus.cc:1067] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Signalling peer 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:12.019876 30695 catalog_manager.cc:5495] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 reported cstate change: term changed from 1 to 2, leader changed from 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130) to bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:12.023190 30951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3657570ad79d43c6a8dce96803512091"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:40302
I20260806 03:59:12.024013 30951 raft_consensus.cc:482] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:12.024695 30951 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:12.033964 30951 raft_consensus.cc:504] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:12.036897 30951 leader_election.cc:288] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 2 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:12.039131 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:12.039505 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:12.039860 31068 raft_consensus.cc:3032] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:12.040277 30846 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:12.040360 31068 raft_consensus.cc:729] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:12.041635 31068 consensus_queue.cc:260] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:12.043357 31068 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:12.044216 30695 catalog_manager.cc:5495] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 reported cstate change: term changed from 1 to 2, leader changed from f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131) to bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:12.050616 30846 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 2.
I20260806 03:59:12.052050 30907 leader_election.cc:302] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:12.053202 31068 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 2.
I20260806 03:59:12.053383 31182 raft_consensus.cc:2781] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:12.054121 31182 raft_consensus.cc:686] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:12.055331 31182 consensus_queue.cc:237] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:12.064393 30695 catalog_manager.cc:5495] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 1 to 2, leader changed from f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131) to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:12.157224 31186 raft_consensus.cc:979] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15: : Instructing follower 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:12.157671 31241 raft_consensus.cc:1067] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Signalling peer 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:12.159132 30951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0a734a4aa10450a905419d5968af802"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:40298
I20260806 03:59:12.159650 30951 raft_consensus.cc:482] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:12.159998 30951 raft_consensus.cc:3037] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:12.164237 30951 raft_consensus.cc:504] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:12.166316 30951 leader_election.cc:288] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 2 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:12.166833 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:12.167421 30846 raft_consensus.cc:3032] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:12.167384 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0a734a4aa10450a905419d5968af802" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:12.167804 30846 raft_consensus.cc:729] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:12.168020 31068 raft_consensus.cc:3037] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:12.168509 30846 consensus_queue.cc:260] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:12.169703 30846 raft_consensus.cc:3037] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:12.174373 31068 raft_consensus.cc:2454] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 2.
I20260806 03:59:12.175698 30906 leader_election.cc:302] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:12.176757 31182 raft_consensus.cc:2781] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:12.177343 31182 raft_consensus.cc:686] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:12.178678 31182 consensus_queue.cc:237] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:12.179616 30846 raft_consensus.cc:2454] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 2.
I20260806 03:59:12.186998 30695 catalog_manager.cc:5495] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 1 to 2, leader changed from bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129) to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:12.291662 31068 raft_consensus.cc:1261] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.292892 31241 consensus_queue.cc:1035] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.302534 30951 raft_consensus.cc:1261] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.304719 31272 consensus_queue.cc:1035] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.354166 31068 raft_consensus.cc:1261] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.355460 31182 consensus_queue.cc:1035] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.364217 30846 raft_consensus.cc:1261] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.366233 31182 consensus_queue.cc:1035] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:59:12.421028 30951 raft_consensus.cc:1261] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.423931 31272 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.435808 31068 raft_consensus.cc:1261] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.438338 31187 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.519949 30951 raft_consensus.cc:1261] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.521195 31187 consensus_queue.cc:1035] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.524688 31068 raft_consensus.cc:1261] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.525828 31187 consensus_queue.cc:1035] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.534690 30846 raft_consensus.cc:1261] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.536064 31146 consensus_queue.cc:1035] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.543826 31068 raft_consensus.cc:1261] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.545401 31146 consensus_queue.cc:1035] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.721280 30846 raft_consensus.cc:1261] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.722519 31146 consensus_queue.cc:1035] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:12.730082 31068 raft_consensus.cc:1261] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:12.731626 31185 consensus_queue.cc:1035] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:14.668766 30646 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260806 03:59:14.669493 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:14.689121 30952 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
mode: GRACEFUL
new_leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
from {username='slave'} at 127.0.0.1:40330
I20260806 03:59:14.689620 30952 raft_consensus.cc:595] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Received request to transfer leadership to TS f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:14.701218 30846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
mode: GRACEFUL
new_leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
from {username='slave'} at 127.0.0.1:55084
I20260806 03:59:14.701676 30846 raft_consensus.cc:595] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Received request to transfer leadership to TS f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:14.702541 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:59:14.702967 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:14.773890 31272 raft_consensus.cc:979] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15: : Instructing follower f4c9090b10ce4e7ead9c0f25ea4eb20b to start an election
I20260806 03:59:14.774296 31287 raft_consensus.cc:1067] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Signalling peer f4c9090b10ce4e7ead9c0f25ea4eb20b to start an election
I20260806 03:59:14.775691 31067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
from {username='slave'} at 127.0.0.1:34262
I20260806 03:59:14.776165 31067 raft_consensus.cc:482] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:14.776413 31067 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:14.780530 31067 raft_consensus.cc:504] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:14.781905 31067 leader_election.cc:288] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 3 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233)
I20260806 03:59:14.782850 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:14.782778 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:14.783475 30846 raft_consensus.cc:3032] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:14.783528 30952 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:14.783842 30846 raft_consensus.cc:729] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:14.784551 30846 consensus_queue.cc:260] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:14.785647 30846 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:14.789959 30952 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 3.
I20260806 03:59:14.791240 31003 leader_election.cc:302] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:14.791414 30846 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 3.
I20260806 03:59:14.792023 31149 raft_consensus.cc:2781] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:14.793324 31149 raft_consensus.cc:686] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 LEADER]: Becoming Leader. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:14.794066 31149 consensus_queue.cc:237] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:14.799867 30695 catalog_manager.cc:5495] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b reported cstate change: term changed from 2 to 3, leader changed from bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129) to f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131). New cstate: current_term: 3 leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:15.257889 31300 raft_consensus.cc:979] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3: : Instructing follower f4c9090b10ce4e7ead9c0f25ea4eb20b to start an election
I20260806 03:59:15.258361 31146 raft_consensus.cc:1067] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Signalling peer f4c9090b10ce4e7ead9c0f25ea4eb20b to start an election
I20260806 03:59:15.259768 31067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
from {username='slave'} at 127.0.0.1:34270
I20260806 03:59:15.260257 31067 raft_consensus.cc:482] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:15.260550 31067 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:15.265273 31067 raft_consensus.cc:504] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:15.266637 31067 leader_election.cc:288] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 2 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:15.267412 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:15.267704 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:15.267989 30846 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:15.268257 30952 raft_consensus.cc:3032] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:15.268569 30952 raft_consensus.cc:729] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:15.269260 30952 consensus_queue.cc:260] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:15.270426 30952 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:15.273883 30846 raft_consensus.cc:2454] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 2.
I20260806 03:59:15.274853 31007 leader_election.cc:302] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:15.275594 31149 raft_consensus.cc:2781] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:15.275954 31149 raft_consensus.cc:686] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 LEADER]: Becoming Leader. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:15.276326 30952 raft_consensus.cc:2454] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 2.
I20260806 03:59:15.276692 31149 consensus_queue.cc:237] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:15.283013 30695 catalog_manager.cc:5495] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b reported cstate change: term changed from 1 to 2, leader changed from 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130) to f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:15.354751 30846 raft_consensus.cc:1261] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Refusing update from remote peer f4c9090b10ce4e7ead9c0f25ea4eb20b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:15.355996 31149 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:15.365113 30952 raft_consensus.cc:1261] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Refusing update from remote peer f4c9090b10ce4e7ead9c0f25ea4eb20b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:15.367295 31306 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:59:15.703852 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:15.708760 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:15.709178 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:15.724364 30952 raft_consensus.cc:1261] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Refusing update from remote peer f4c9090b10ce4e7ead9c0f25ea4eb20b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:15.725525 31306 consensus_queue.cc:1035] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:15.734437 30847 raft_consensus.cc:1261] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Refusing update from remote peer f4c9090b10ce4e7ead9c0f25ea4eb20b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:15.736111 31149 consensus_queue.cc:1035] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:16.710011 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:16.715161 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:16.715557 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:17.746220 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:17.760242 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:17.760874 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:59:18.728688 30747 debug-util.cc:398] Leaking SignalData structure 0x7b0800060060 after lost signal to thread 30649
W20260806 03:59:18.729449 30747 debug-util.cc:398] Leaking SignalData structure 0x7b0800060740 after lost signal to thread 30751
W20260806 03:59:18.730159 30747 debug-util.cc:398] Leaking SignalData structure 0x7b0800153960 after lost signal to thread 30883
W20260806 03:59:18.730813 30747 debug-util.cc:398] Leaking SignalData structure 0x7b080010cb20 after lost signal to thread 30983
W20260806 03:59:18.731417 30747 debug-util.cc:398] Leaking SignalData structure 0x7b08001fda60 after lost signal to thread 31104
I20260806 03:59:18.761765 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:18.766419 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:18.766788 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:19.767608 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:19.774560 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:19.775072 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:20.775971 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:20.780675 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:20.781030 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:21.781785 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:21.786327 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:21.786666 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:22.787441 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:22.791850 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:22.792181 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:23.792964 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:23.797706 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:23.798049 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:24.823459 30646 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260806 03:59:24.827836 31068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
mode: GRACEFUL
new_leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:34274
I20260806 03:59:24.828542 31068 raft_consensus.cc:595] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 LEADER]: Received request to transfer leadership to TS 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:24.833141 30847 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
mode: GRACEFUL
new_leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:55080
I20260806 03:59:24.833640 30847 raft_consensus.cc:595] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Received request to transfer leadership to TS 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:24.837687 30847 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
mode: GRACEFUL
new_leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:55080
I20260806 03:59:24.838122 30847 raft_consensus.cc:595] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Received request to transfer leadership to TS 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:24.842013 30847 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14f89048e201458ebc05796417c0ae31"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
mode: GRACEFUL
new_leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:55080
I20260806 03:59:24.842451 30847 raft_consensus.cc:595] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Received request to transfer leadership to TS 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:24.853207 31068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
mode: GRACEFUL
new_leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:34274
I20260806 03:59:24.853781 31068 raft_consensus.cc:595] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 LEADER]: Received request to transfer leadership to TS 098e0e271e91425cafa13cdd4f19f8b3
I20260806 03:59:24.992589 31337 raft_consensus.cc:979] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15: : Instructing follower 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:24.993142 31323 raft_consensus.cc:1067] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Signalling peer 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:24.994705 30951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:40298
I20260806 03:59:24.995162 30951 raft_consensus.cc:482] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:24.995443 30951 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.002080 30951 raft_consensus.cc:504] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.004570 30951 leader_election.cc:288] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 3 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:25.005455 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:25.005686 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:25.006388 30847 raft_consensus.cc:3032] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:25.006464 31068 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.006796 30847 raft_consensus.cc:729] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:25.008000 30847 consensus_queue.cc:260] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.009747 30847 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.015524 31068 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 3.
I20260806 03:59:25.017426 30906 leader_election.cc:302] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:25.018635 30847 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 3.
I20260806 03:59:25.019760 31335 raft_consensus.cc:2781] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:25.020670 31335 raft_consensus.cc:686] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:25.021821 31335 consensus_queue.cc:237] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.032624 30695 catalog_manager.cc:5495] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 2 to 3, leader changed from bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129) to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 3 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:25.062204 31337 raft_consensus.cc:979] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15: : Instructing follower 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:25.062824 31323 raft_consensus.cc:1067] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Signalling peer 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:25.064952 30951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:40298
I20260806 03:59:25.065596 30951 raft_consensus.cc:482] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:25.065964 30951 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.073158 30951 raft_consensus.cc:504] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.075182 30951 leader_election.cc:288] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 3 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:25.076102 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:25.076798 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:25.076943 30847 raft_consensus.cc:3032] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:25.077522 31068 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.077466 30847 raft_consensus.cc:729] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:25.078792 30847 consensus_queue.cc:260] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.080377 30847 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.083869 31068 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 3.
I20260806 03:59:25.085309 30906 leader_election.cc:302] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:25.086387 31335 raft_consensus.cc:2781] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:25.086843 31335 raft_consensus.cc:686] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:25.087885 30847 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 3.
I20260806 03:59:25.087849 31335 consensus_queue.cc:237] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.096433 30695 catalog_manager.cc:5495] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 2 to 3, leader changed from bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129) to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 3 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:25.104555 31337 raft_consensus.cc:979] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15: : Instructing follower 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:25.105074 31323 raft_consensus.cc:1067] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Signalling peer 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:25.106865 30951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14f89048e201458ebc05796417c0ae31"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:40298
I20260806 03:59:25.107493 30951 raft_consensus.cc:482] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:25.107913 30951 raft_consensus.cc:3037] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:25.114818 30951 raft_consensus.cc:504] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.117167 30951 leader_election.cc:288] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 2 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:25.118037 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f89048e201458ebc05796417c0ae31" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:25.118548 30847 raft_consensus.cc:3032] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:25.118597 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f89048e201458ebc05796417c0ae31" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:25.118793 30847 raft_consensus.cc:729] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:25.119114 31068 raft_consensus.cc:3037] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:25.119472 30847 consensus_queue.cc:260] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.120425 30847 raft_consensus.cc:3037] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:25.124714 30847 raft_consensus.cc:2454] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 2.
I20260806 03:59:25.125607 31068 raft_consensus.cc:2454] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 2.
I20260806 03:59:25.126233 30907 leader_election.cc:302] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:25.127290 31335 raft_consensus.cc:2781] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:25.127790 31335 raft_consensus.cc:686] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:25.128958 31335 consensus_queue.cc:237] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.136852 30695 catalog_manager.cc:5495] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 1 to 2, leader changed from bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129) to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:25.141664 31336 raft_consensus.cc:979] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b: : Instructing follower 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:25.142686 31290 raft_consensus.cc:1067] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 LEADER]: Signalling peer 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:25.144651 30951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:40302
I20260806 03:59:25.145390 30951 raft_consensus.cc:482] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:25.145885 30951 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:25.150285 31336 raft_consensus.cc:979] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b: : Instructing follower 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:25.150900 31290 raft_consensus.cc:1067] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 LEADER]: Signalling peer 098e0e271e91425cafa13cdd4f19f8b3 to start an election
I20260806 03:59:25.153270 30952 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
from {username='slave'} at 127.0.0.1:40302
I20260806 03:59:25.153791 30952 raft_consensus.cc:482] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:25.154127 30952 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.153962 30951 raft_consensus.cc:504] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:25.155864 30951 leader_election.cc:288] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 4 election: Requested vote from peers f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679), bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233)
I20260806 03:59:25.157071 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:25.157550 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:25.157992 30847 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:25.157801 31068 raft_consensus.cc:3032] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:59:25.158389 31068 raft_consensus.cc:729] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:25.159618 31068 consensus_queue.cc:260] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:25.161250 31068 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:25.161754 30952 raft_consensus.cc:504] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.162577 30847 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 4.
I20260806 03:59:25.163355 30907 leader_election.cc:302] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:25.163731 30952 leader_election.cc:288] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 3 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:25.165215 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:25.165360 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "098e0e271e91425cafa13cdd4f19f8b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:25.165773 31067 raft_consensus.cc:3032] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:25.165856 30847 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.166100 31067 raft_consensus.cc:729] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:25.166859 31067 consensus_queue.cc:260] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.167881 31067 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:25.168798 31068 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 4.
I20260806 03:59:25.169221 31335 raft_consensus.cc:2781] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:59:25.172550 31335 raft_consensus.cc:686] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:25.173842 30847 raft_consensus.cc:2454] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 3.
I20260806 03:59:25.173810 31335 consensus_queue.cc:237] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:25.175046 30907 leader_election.cc:302] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098e0e271e91425cafa13cdd4f19f8b3, bde9de7562e24ecdb40f72bf3f753e15; no voters:
I20260806 03:59:25.175691 31067 raft_consensus.cc:2454] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 098e0e271e91425cafa13cdd4f19f8b3 in term 3.
I20260806 03:59:25.176710 31348 raft_consensus.cc:2781] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:25.177407 31348 raft_consensus.cc:686] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Becoming Leader. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:25.178799 31348 consensus_queue.cc:237] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:25.184599 30695 catalog_manager.cc:5495] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 3 to 4, leader changed from f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131) to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 4 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:25.196642 30695 catalog_manager.cc:5495] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 reported cstate change: term changed from 2 to 3, leader changed from f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131) to 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130). New cstate: current_term: 3 leader_uuid: "098e0e271e91425cafa13cdd4f19f8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:25.414382 31067 raft_consensus.cc:1261] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:25.415629 31348 consensus_queue.cc:1035] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:59:25.432551 30846 raft_consensus.cc:1261] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:25.434361 31348 consensus_queue.cc:1035] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:59:25.587606 30846 raft_consensus.cc:1261] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:25.589273 31338 consensus_queue.cc:1035] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:59:25.601404 31067 raft_consensus.cc:1261] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:25.602947 31338 consensus_queue.cc:1035] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:59:25.625999 31067 raft_consensus.cc:1261] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:25.627251 31338 consensus_queue.cc:1035] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:25.638924 30846 raft_consensus.cc:1261] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:25.640429 31348 consensus_queue.cc:1035] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:25.691282 31068 raft_consensus.cc:1261] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:25.691715 30846 raft_consensus.cc:1261] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:25.692989 31348 consensus_queue.cc:1035] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:25.693785 31335 consensus_queue.cc:1035] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:25.694523 30846 raft_consensus.cc:1261] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:25.696118 31335 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:25.719739 31068 raft_consensus.cc:1261] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Refusing update from remote peer 098e0e271e91425cafa13cdd4f19f8b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:25.721452 31338 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:27.884363 30646 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260806 03:59:27.885083 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:27.891566 30952 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
mode: GRACEFUL
new_leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:40330
I20260806 03:59:27.892236 30952 raft_consensus.cc:595] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Received request to transfer leadership to TS bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:27.894660 30952 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3657570ad79d43c6a8dce96803512091"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
mode: GRACEFUL
new_leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:40330
I20260806 03:59:27.895134 30952 raft_consensus.cc:595] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Received request to transfer leadership to TS bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:27.897197 30952 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
mode: GRACEFUL
new_leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
from {username='slave'} at 127.0.0.1:40330
I20260806 03:59:27.897578 30952 raft_consensus.cc:595] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 LEADER]: Received request to transfer leadership to TS f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:27.899531 30952 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
mode: GRACEFUL
new_leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
from {username='slave'} at 127.0.0.1:40330
I20260806 03:59:27.899902 30952 raft_consensus.cc:595] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Received request to transfer leadership to TS f4c9090b10ce4e7ead9c0f25ea4eb20b
I20260806 03:59:27.901700 30952 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1"
dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
mode: GRACEFUL
new_leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:40330
I20260806 03:59:27.902086 30952 raft_consensus.cc:595] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Received request to transfer leadership to TS bde9de7562e24ecdb40f72bf3f753e15
I20260806 03:59:27.902839 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 03:59:27.903272 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:27.966136 31348 raft_consensus.cc:979] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3: : Instructing follower bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:27.966733 31377 raft_consensus.cc:1067] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Signalling peer bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:27.968410 30846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:55052
I20260806 03:59:27.968973 30846 raft_consensus.cc:482] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:27.969071 31348 raft_consensus.cc:979] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3: : Instructing follower bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:27.969389 30846 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:27.969568 31377 raft_consensus.cc:1067] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Signalling peer bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:27.971232 30844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3657570ad79d43c6a8dce96803512091"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:55052
I20260806 03:59:27.971768 30844 raft_consensus.cc:482] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:27.972122 30844 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:27.977546 30846 raft_consensus.cc:504] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:27.978916 30844 raft_consensus.cc:504] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:27.980400 30846 leader_election.cc:288] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 4 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:27.980917 30844 leader_election.cc:288] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 3 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:27.983054 31067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:27.983801 31067 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:27.983160 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:27.984663 31068 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:27.990585 31067 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 3.
I20260806 03:59:27.990792 30951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3657570ad79d43c6a8dce96803512091" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:27.991380 30951 raft_consensus.cc:3032] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:27.991681 30951 raft_consensus.cc:729] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:27.992125 30795 leader_election.cc:302] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:27.992414 30951 consensus_queue.cc:260] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:27.994231 30951 raft_consensus.cc:3037] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:27.990749 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb5e025e0c43cf879ba801e672a0f1" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:27.995190 30952 raft_consensus.cc:3032] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:59:27.995513 30952 raft_consensus.cc:729] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:27.995612 31068 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 4.
I20260806 03:59:27.996490 30952 consensus_queue.cc:260] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:27.997949 30952 raft_consensus.cc:3037] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:27.999501 30951 raft_consensus.cc:2454] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 3.
I20260806 03:59:28.000974 31380 raft_consensus.cc:2781] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:28.000909 30795 leader_election.cc:302] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:28.002153 31380 raft_consensus.cc:686] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 LEADER]: Becoming Leader. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:28.003365 31381 raft_consensus.cc:2781] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:59:28.003098 31380 consensus_queue.cc:237] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:28.003857 31381 raft_consensus.cc:686] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 4 LEADER]: Becoming Leader. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:28.005015 30952 raft_consensus.cc:2454] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 4.
I20260806 03:59:28.004930 31381 consensus_queue.cc:237] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:28.012826 30698 catalog_manager.cc:5495] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 reported cstate change: term changed from 2 to 3, leader changed from 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130) to bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:28.029179 30698 catalog_manager.cc:5495] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 reported cstate change: term changed from 3 to 4, leader changed from 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130) to bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129). New cstate: current_term: 4 leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:28.049417 31348 raft_consensus.cc:979] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3: : Instructing follower bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:28.049841 31377 raft_consensus.cc:1067] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Signalling peer bde9de7562e24ecdb40f72bf3f753e15 to start an election
I20260806 03:59:28.051159 30844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d"
dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
from {username='slave'} at 127.0.0.1:55052
I20260806 03:59:28.051668 30844 raft_consensus.cc:482] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:28.051963 30844 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:28.056375 30844 raft_consensus.cc:504] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:28.057822 30844 leader_election.cc:288] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 4 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131:39679)
I20260806 03:59:28.058611 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:28.058931 31068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33400b2f7d8f4103ac04c30fa55fb96d" candidate_uuid: "bde9de7562e24ecdb40f72bf3f753e15" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
I20260806 03:59:28.059170 30952 raft_consensus.cc:3032] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:59:28.059419 31068 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:28.059532 30952 raft_consensus.cc:729] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:28.060302 30952 consensus_queue.cc:260] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:28.061455 30952 raft_consensus.cc:3037] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:28.065416 31068 raft_consensus.cc:2454] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 4.
I20260806 03:59:28.066390 30795 leader_election.cc:302] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:28.066960 30952 raft_consensus.cc:2454] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde9de7562e24ecdb40f72bf3f753e15 in term 4.
I20260806 03:59:28.067044 31381 raft_consensus.cc:2781] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:59:28.067571 31381 raft_consensus.cc:686] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 4 LEADER]: Becoming Leader. State: Replica: bde9de7562e24ecdb40f72bf3f753e15, State: Running, Role: LEADER
I20260806 03:59:28.068531 31381 consensus_queue.cc:237] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:28.075340 30698 catalog_manager.cc:5495] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 reported cstate change: term changed from 3 to 4, leader changed from 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130) to bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129). New cstate: current_term: 4 leader_uuid: "bde9de7562e24ecdb40f72bf3f753e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:28.099890 31348 raft_consensus.cc:979] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3: : Instructing follower f4c9090b10ce4e7ead9c0f25ea4eb20b to start an election
I20260806 03:59:28.100499 31378 raft_consensus.cc:1067] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 LEADER]: Signalling peer f4c9090b10ce4e7ead9c0f25ea4eb20b to start an election
I20260806 03:59:28.101980 31068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
from {username='slave'} at 127.0.0.1:34270
I20260806 03:59:28.102404 31068 raft_consensus.cc:482] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:28.102684 31068 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:59:28.106840 31068 raft_consensus.cc:504] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:28.108659 31068 leader_election.cc:288] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 5 election: Requested vote from peers 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147), bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233)
I20260806 03:59:28.109427 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:28.109699 30844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a622c056db4064ba7658fc3db5315d" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:28.109993 30952 raft_consensus.cc:3032] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:59:28.110226 30844 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:59:28.110322 30952 raft_consensus.cc:729] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:28.111054 30952 consensus_queue.cc:260] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:28.112277 30952 raft_consensus.cc:3037] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:59:28.116755 30844 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 5.
I20260806 03:59:28.118047 31007 leader_election.cc:302] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:28.118191 30952 raft_consensus.cc:2454] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 5.
I20260806 03:59:28.118824 31290 raft_consensus.cc:2781] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:59:28.119908 31290 raft_consensus.cc:686] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 5 LEADER]: Becoming Leader. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:28.120594 31290 consensus_queue.cc:237] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } }
I20260806 03:59:28.126732 30698 catalog_manager.cc:5495] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b reported cstate change: term changed from 4 to 5, leader changed from 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130) to f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131). New cstate: current_term: 5 leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:28.155915 31377 raft_consensus.cc:979] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3: : Instructing follower f4c9090b10ce4e7ead9c0f25ea4eb20b to start an election
I20260806 03:59:28.156373 31378 raft_consensus.cc:1067] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Signalling peer f4c9090b10ce4e7ead9c0f25ea4eb20b to start an election
I20260806 03:59:28.157688 31068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc"
dest_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b"
from {username='slave'} at 127.0.0.1:34270
I20260806 03:59:28.158083 31068 raft_consensus.cc:482] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:28.158293 31068 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:28.162272 31068 raft_consensus.cc:504] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:28.163520 31068 leader_election.cc:288] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 4 election: Requested vote from peers bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233), 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147)
I20260806 03:59:28.164356 30844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bde9de7562e24ecdb40f72bf3f753e15"
I20260806 03:59:28.164554 30952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5486ddbb7fc34b0da6220b44d27f0ecc" candidate_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "098e0e271e91425cafa13cdd4f19f8b3"
I20260806 03:59:28.164949 30844 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:28.165093 30952 raft_consensus.cc:3032] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:59:28.165406 30952 raft_consensus.cc:729] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 098e0e271e91425cafa13cdd4f19f8b3, State: Running, Role: LEADER
I20260806 03:59:28.166006 30952 consensus_queue.cc:260] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:28.166908 30952 raft_consensus.cc:3037] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:28.170029 30844 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 4.
I20260806 03:59:28.170816 30952 raft_consensus.cc:2454] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c9090b10ce4e7ead9c0f25ea4eb20b in term 4.
I20260806 03:59:28.170840 31007 leader_election.cc:302] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bde9de7562e24ecdb40f72bf3f753e15, f4c9090b10ce4e7ead9c0f25ea4eb20b; no voters:
I20260806 03:59:28.171614 31290 raft_consensus.cc:2781] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:59:28.171952 31290 raft_consensus.cc:686] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 LEADER]: Becoming Leader. State: Replica: f4c9090b10ce4e7ead9c0f25ea4eb20b, State: Running, Role: LEADER
I20260806 03:59:28.172613 31290 consensus_queue.cc:237] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } }
I20260806 03:59:28.178207 30698 catalog_manager.cc:5495] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b reported cstate change: term changed from 3 to 4, leader changed from 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130) to f4c9090b10ce4e7ead9c0f25ea4eb20b (127.29.237.131). New cstate: current_term: 4 leader_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:28.399794 30952 raft_consensus.cc:1261] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:28.401077 31381 consensus_queue.cc:1035] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:28.412467 31068 raft_consensus.cc:1261] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:28.413764 31380 consensus_queue.cc:1035] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:28.458300 30952 raft_consensus.cc:1261] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:28.459501 31381 consensus_queue.cc:1035] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:28.460604 30952 raft_consensus.cc:1261] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:28.461688 31381 consensus_queue.cc:1035] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:28.471297 31068 raft_consensus.cc:1261] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:28.471935 31067 raft_consensus.cc:1261] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Refusing update from remote peer bde9de7562e24ecdb40f72bf3f753e15: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:28.474328 31380 consensus_queue.cc:1035] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:28.474931 31383 consensus_queue.cc:1035] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c9090b10ce4e7ead9c0f25ea4eb20b" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:59:28.669679 30951 raft_consensus.cc:1261] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Refusing update from remote peer f4c9090b10ce4e7ead9c0f25ea4eb20b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:28.670785 31290 consensus_queue.cc:1035] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:28.678141 30951 raft_consensus.cc:1261] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 5 FOLLOWER]: Refusing update from remote peer f4c9090b10ce4e7ead9c0f25ea4eb20b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 03:59:28.679358 30844 raft_consensus.cc:1261] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Refusing update from remote peer f4c9090b10ce4e7ead9c0f25ea4eb20b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:28.679729 31290 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "098e0e271e91425cafa13cdd4f19f8b3" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:59:28.680768 31423 consensus_queue.cc:1035] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:28.697925 30844 raft_consensus.cc:1261] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 5 FOLLOWER]: Refusing update from remote peer f4c9090b10ce4e7ead9c0f25ea4eb20b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 03:59:28.699715 31423 consensus_queue.cc:1035] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde9de7562e24ecdb40f72bf3f753e15" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:59:28.904147 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:28.909153 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:28.909495 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:30.017221 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:30.022356 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:30.022790 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:59:30.651896 31098 debug-util.cc:398] Leaking SignalData structure 0x7b0800140fe0 after lost signal to thread 30649
W20260806 03:59:30.652683 31098 debug-util.cc:398] Leaking SignalData structure 0x7b0800140900 after lost signal to thread 30751
W20260806 03:59:30.653424 31098 debug-util.cc:398] Leaking SignalData structure 0x7b080010cde0 after lost signal to thread 30883
W20260806 03:59:30.654093 31098 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6420 after lost signal to thread 30983
W20260806 03:59:30.654717 31098 debug-util.cc:398] Leaking SignalData structure 0x7b0800087e80 after lost signal to thread 31104
I20260806 03:59:31.023729 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:31.031981 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:31.032716 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:32.033628 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:32.038717 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:32.039180 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:33.054975 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:33.059845 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:33.060248 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:59:33.160033 30978 debug-util.cc:398] Leaking SignalData structure 0x7b0800132060 after lost signal to thread 30649
W20260806 03:59:33.161087 30978 debug-util.cc:398] Leaking SignalData structure 0x7b0800132740 after lost signal to thread 30751
W20260806 03:59:33.162040 30978 debug-util.cc:398] Leaking SignalData structure 0x7b08001d8a80 after lost signal to thread 30883
W20260806 03:59:33.162843 30978 debug-util.cc:398] Leaking SignalData structure 0x7b0800140880 after lost signal to thread 30983
W20260806 03:59:33.163806 30978 debug-util.cc:398] Leaking SignalData structure 0x7b0800196380 after lost signal to thread 31104
I20260806 03:59:34.061160 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:34.066260 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:34.066634 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:35.067330 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:35.072033 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:35.072431 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:36.073285 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:36.078014 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:36.078377 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:37.079159 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:37.083835 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:37.084256 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:38.111703 30646 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260806 03:59:38.112326 30646 tablet_server.cc:178] TabletServer@127.29.237.129:0 shutting down...
I20260806 03:59:38.139102 30646 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:59:38.139833 30646 tablet_replica.cc:330] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.140875 30646 raft_consensus.cc:2229] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.141700 30646 raft_consensus.cc:2258] T 5d757fe41bf345f8a9ba96f6e4160dd1 P bde9de7562e24ecdb40f72bf3f753e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.144024 30646 tablet_replica.cc:330] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.144708 30646 raft_consensus.cc:2229] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.145655 30646 raft_consensus.cc:2258] T 3657570ad79d43c6a8dce96803512091 P bde9de7562e24ecdb40f72bf3f753e15 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.147888 30646 tablet_replica.cc:330] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.148445 30646 raft_consensus.cc:2229] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.148922 30646 raft_consensus.cc:2258] T 49a622c056db4064ba7658fc3db5315d P bde9de7562e24ecdb40f72bf3f753e15 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.150532 30646 tablet_replica.cc:330] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.151069 30646 raft_consensus.cc:2229] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.151486 30646 raft_consensus.cc:2258] T 5486ddbb7fc34b0da6220b44d27f0ecc P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.153141 30646 tablet_replica.cc:330] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.153580 30646 raft_consensus.cc:2229] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.154367 30646 raft_consensus.cc:2258] T b0eb5e025e0c43cf879ba801e672a0f1 P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.155864 30646 tablet_replica.cc:330] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.156351 30646 raft_consensus.cc:2229] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.156734 30646 raft_consensus.cc:2258] T d0a734a4aa10450a905419d5968af802 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.158187 30646 tablet_replica.cc:330] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.158607 30646 raft_consensus.cc:2229] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.158985 30646 raft_consensus.cc:2258] T 14f89048e201458ebc05796417c0ae31 P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.160701 30646 tablet_replica.cc:330] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.161128 30646 raft_consensus.cc:2229] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.161998 30646 raft_consensus.cc:2258] T 33400b2f7d8f4103ac04c30fa55fb96d P bde9de7562e24ecdb40f72bf3f753e15 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260806 03:59:38.163740 30907 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260806 03:59:38.164255 30646 tablet_replica.cc:330] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15: stopping tablet replica
I20260806 03:59:38.164867 30646 raft_consensus.cc:2229] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.165385 30646 raft_consensus.cc:2258] T 63c3ef8772aa477aadf447386445353b P bde9de7562e24ecdb40f72bf3f753e15 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260806 03:59:38.167026 30907 consensus_peers.cc:487] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 -> Peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Couldn't send request to peer bde9de7562e24ecdb40f72bf3f753e15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:59:38.256616 31007 consensus_peers.cc:487] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b -> Peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Couldn't send request to peer bde9de7562e24ecdb40f72bf3f753e15. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.129:40233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:59:38.257119 30646 tablet_server.cc:194] TabletServer@127.29.237.129:0 shutdown complete.
I20260806 03:59:38.274112 30646 tablet_server.cc:178] TabletServer@127.29.237.130:0 shutting down...
W20260806 03:59:38.281188 31003 consensus_peers.cc:487] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b -> Peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Couldn't send request to peer 098e0e271e91425cafa13cdd4f19f8b3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:59:38.289067 30907 consensus_peers.cc:487] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 -> Peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Couldn't send request to peer bde9de7562e24ecdb40f72bf3f753e15. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.129:40233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:59:38.310694 30646 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:59:38.311403 30646 tablet_replica.cc:330] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.312273 30646 raft_consensus.cc:2229] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.312727 30646 raft_consensus.cc:2258] T 5d757fe41bf345f8a9ba96f6e4160dd1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.314502 30646 tablet_replica.cc:330] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.314935 30646 raft_consensus.cc:2229] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.315680 30646 raft_consensus.cc:2258] T 63c3ef8772aa477aadf447386445353b P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.317307 30646 tablet_replica.cc:330] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.318002 30646 raft_consensus.cc:2229] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.318425 30646 raft_consensus.cc:2258] T 33400b2f7d8f4103ac04c30fa55fb96d P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.320114 30646 tablet_replica.cc:330] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.320580 30646 raft_consensus.cc:2229] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.320987 30646 raft_consensus.cc:2258] T 5486ddbb7fc34b0da6220b44d27f0ecc P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.322530 30646 tablet_replica.cc:330] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.322954 30646 raft_consensus.cc:2229] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.323395 30646 raft_consensus.cc:2258] T 49a622c056db4064ba7658fc3db5315d P 098e0e271e91425cafa13cdd4f19f8b3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.324910 30646 tablet_replica.cc:330] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.325328 30646 raft_consensus.cc:2229] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.325701 30646 raft_consensus.cc:2258] T 3657570ad79d43c6a8dce96803512091 P 098e0e271e91425cafa13cdd4f19f8b3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.327183 30646 tablet_replica.cc:330] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.327600 30646 raft_consensus.cc:2229] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.328423 30646 raft_consensus.cc:2258] T 14f89048e201458ebc05796417c0ae31 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.329952 30646 tablet_replica.cc:330] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.330363 30646 raft_consensus.cc:2229] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.331115 30646 raft_consensus.cc:2258] T d0a734a4aa10450a905419d5968af802 P 098e0e271e91425cafa13cdd4f19f8b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.332691 30646 tablet_replica.cc:330] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3: stopping tablet replica
I20260806 03:59:38.333098 30646 raft_consensus.cc:2229] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.333472 30646 raft_consensus.cc:2258] T b0eb5e025e0c43cf879ba801e672a0f1 P 098e0e271e91425cafa13cdd4f19f8b3 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260806 03:59:38.352384 31007 consensus_peers.cc:487] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b -> Peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Couldn't send request to peer bde9de7562e24ecdb40f72bf3f753e15. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.129:40233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260806 03:59:38.364853 31007 consensus_peers.cc:487] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b -> Peer bde9de7562e24ecdb40f72bf3f753e15 (127.29.237.129:40233): Couldn't send request to peer bde9de7562e24ecdb40f72bf3f753e15. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.129:40233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:59:38.379211 30646 tablet_server.cc:194] TabletServer@127.29.237.130:0 shutdown complete.
I20260806 03:59:38.398039 30646 tablet_server.cc:178] TabletServer@127.29.237.131:0 shutting down...
W20260806 03:59:38.405010 31003 consensus_peers.cc:487] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b -> Peer 098e0e271e91425cafa13cdd4f19f8b3 (127.29.237.130:34147): Couldn't send request to peer 098e0e271e91425cafa13cdd4f19f8b3. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.130:34147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:59:38.425982 30646 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:59:38.426717 30646 tablet_replica.cc:330] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.427345 30646 raft_consensus.cc:2229] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.428334 30646 raft_consensus.cc:2258] T 5486ddbb7fc34b0da6220b44d27f0ecc P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.430148 30646 tablet_replica.cc:330] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.430652 30646 raft_consensus.cc:2229] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.431484 30646 raft_consensus.cc:2258] T 5d757fe41bf345f8a9ba96f6e4160dd1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.433254 30646 tablet_replica.cc:330] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.433745 30646 raft_consensus.cc:2229] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.434219 30646 raft_consensus.cc:2258] T 63c3ef8772aa477aadf447386445353b P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.435963 30646 tablet_replica.cc:330] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.436503 30646 raft_consensus.cc:2229] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.436941 30646 raft_consensus.cc:2258] T 33400b2f7d8f4103ac04c30fa55fb96d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.438539 30646 tablet_replica.cc:330] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.439003 30646 raft_consensus.cc:2229] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 5 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.439810 30646 raft_consensus.cc:2258] T 49a622c056db4064ba7658fc3db5315d P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.441421 30646 tablet_replica.cc:330] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.441869 30646 raft_consensus.cc:2229] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.442268 30646 raft_consensus.cc:2258] T 3657570ad79d43c6a8dce96803512091 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.443806 30646 tablet_replica.cc:330] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.444265 30646 raft_consensus.cc:2229] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.444759 30646 raft_consensus.cc:2258] T b0eb5e025e0c43cf879ba801e672a0f1 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.446440 30646 tablet_replica.cc:330] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.446898 30646 raft_consensus.cc:2229] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.447309 30646 raft_consensus.cc:2258] T d0a734a4aa10450a905419d5968af802 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.448891 30646 tablet_replica.cc:330] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b: stopping tablet replica
I20260806 03:59:38.449303 30646 raft_consensus.cc:2229] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:59:38.449680 30646 raft_consensus.cc:2258] T 14f89048e201458ebc05796417c0ae31 P f4c9090b10ce4e7ead9c0f25ea4eb20b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.492964 30646 tablet_server.cc:194] TabletServer@127.29.237.131:0 shutdown complete.
I20260806 03:59:38.510090 30646 master.cc:537] Master@127.29.237.190:35895 shutting down...
I20260806 03:59:38.836552 30646 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:59:38.837157 30646 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:59:38.837457 30646 tablet_replica.cc:330] T 00000000000000000000000000000000 P b8e734a12cb14d29818b0f4f32a9beb0: stopping tablet replica
I20260806 03:59:38.870802 30646 master.cc:552] Master@127.29.237.190:35895 shutdown complete.
098e0e271e91425cafa13cdd4f19f8b3 4
bde9de7562e24ecdb40f72bf3f753e15 4
f4c9090b10ce4e7ead9c0f25ea4eb20b 1
098e0e271e91425cafa13cdd4f19f8b3 3
bde9de7562e24ecdb40f72bf3f753e15 3
f4c9090b10ce4e7ead9c0f25ea4eb20b 3
098e0e271e91425cafa13cdd4f19f8b3 8
bde9de7562e24ecdb40f72bf3f753e15 0
f4c9090b10ce4e7ead9c0f25ea4eb20b 1
098e0e271e91425cafa13cdd4f19f8b3 3
bde9de7562e24ecdb40f72bf3f753e15 3
f4c9090b10ce4e7ead9c0f25ea4eb20b 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31597 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260806 03:59:38.911239 30646 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.237.190:38723
I20260806 03:59:38.912328 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:59:38.917857 31452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:38.918066 31453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:38.919121 31458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:38.920354 30646 server_base.cc:1034] running on GCE node
I20260806 03:59:38.921351 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:59:38.921541 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:59:38.921649 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988778921641 us; error 0 us; skew 500 ppm
I20260806 03:59:38.922078 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:59:38.927390 30646 webserver.cc:460] Webserver started at http://127.29.237.190:37571/ using document root <none> and password file <none>
I20260806 03:59:38.927850 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 03:59:38.927999 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:59:38.928313 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:59:38.929571 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/master-0-root/instance:
uuid: "78614853f6fd40a7a8a2360261e5cca3"
format_stamp: "Formatted at 2026-08-06 03:59:38 on dist-test-slave-lgr5"
I20260806 03:59:38.933967 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 03:59:38.937196 31466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:38.937872 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 03:59:38.938133 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/master-0-root
uuid: "78614853f6fd40a7a8a2360261e5cca3"
format_stamp: "Formatted at 2026-08-06 03:59:38 on dist-test-slave-lgr5"
I20260806 03:59:38.938372 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:59:38.962909 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:59:38.964394 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:59:39.001900 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.190:38723
I20260806 03:59:39.002003 31551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.190:38723 every 8 connection(s)
I20260806 03:59:39.005755 31553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.014948 31553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3: Bootstrap starting.
I20260806 03:59:39.019433 31553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.023947 31553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3: No bootstrap required, opened a new log
I20260806 03:59:39.026223 31553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78614853f6fd40a7a8a2360261e5cca3" member_type: VOTER }
I20260806 03:59:39.026678 31553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.026880 31553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78614853f6fd40a7a8a2360261e5cca3, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.027400 31553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78614853f6fd40a7a8a2360261e5cca3" member_type: VOTER }
I20260806 03:59:39.027822 31553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:59:39.028008 31553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:59:39.028234 31553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:39.031903 31553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78614853f6fd40a7a8a2360261e5cca3" member_type: VOTER }
I20260806 03:59:39.032446 31553 leader_election.cc:302] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78614853f6fd40a7a8a2360261e5cca3; no voters:
I20260806 03:59:39.033640 31553 leader_election.cc:288] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:59:39.034000 31557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:39.035320 31557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 1 LEADER]: Becoming Leader. State: Replica: 78614853f6fd40a7a8a2360261e5cca3, State: Running, Role: LEADER
I20260806 03:59:39.035971 31557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78614853f6fd40a7a8a2360261e5cca3" member_type: VOTER }
I20260806 03:59:39.036578 31553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:59:39.041523 31560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78614853f6fd40a7a8a2360261e5cca3. Latest consensus state: current_term: 1 leader_uuid: "78614853f6fd40a7a8a2360261e5cca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78614853f6fd40a7a8a2360261e5cca3" member_type: VOTER } }
I20260806 03:59:39.041857 31558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78614853f6fd40a7a8a2360261e5cca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78614853f6fd40a7a8a2360261e5cca3" member_type: VOTER } }
I20260806 03:59:39.042197 31560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:59:39.042356 31558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [sys.catalog]: This master's current role is: LEADER
I20260806 03:59:39.049161 31572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:59:39.056365 31572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:59:39.057345 30646 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 03:59:39.064697 31572 catalog_manager.cc:1347] Generated new cluster ID: 6f31da3e1e46474b8eacdcda974e76b5
I20260806 03:59:39.064975 31572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:59:39.086519 31572 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:59:39.088110 31572 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:59:39.102686 31572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3: Generated new TSK 0
I20260806 03:59:39.103327 31572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:59:39.123910 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:59:39.130908 31592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:39.130110 31589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:39.132277 31595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:39.132864 30646 server_base.cc:1034] running on GCE node
I20260806 03:59:39.133810 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:59:39.134007 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:59:39.134152 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988779134136 us; error 0 us; skew 500 ppm
I20260806 03:59:39.134646 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:59:39.137130 30646 webserver.cc:460] Webserver started at http://127.29.237.129:44079/ using document root <none> and password file <none>
I20260806 03:59:39.137621 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 03:59:39.137807 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:59:39.138069 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:59:39.139300 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-0-root/instance:
uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
format_stamp: "Formatted at 2026-08-06 03:59:39 on dist-test-slave-lgr5"
I20260806 03:59:39.143764 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 03:59:39.146975 31604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.148409 30646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:59:39.148653 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-0-root
uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
format_stamp: "Formatted at 2026-08-06 03:59:39 on dist-test-slave-lgr5"
I20260806 03:59:39.148893 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:59:39.166605 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:59:39.167798 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:59:39.169248 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:59:39.171391 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:59:39.171559 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.171729 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:59:39.171865 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.209877 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.129:37197
I20260806 03:59:39.209955 31701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.129:37197 every 8 connection(s)
I20260806 03:59:39.214694 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:59:39.222131 31709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:39.223943 31710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:39.225970 31712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:39.226092 30646 server_base.cc:1034] running on GCE node
I20260806 03:59:39.227433 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:59:39.227676 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:59:39.227866 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988779227850 us; error 0 us; skew 500 ppm
I20260806 03:59:39.228588 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:59:39.229671 31702 heartbeater.cc:347] Connected to a master server at 127.29.237.190:38723
I20260806 03:59:39.230016 31702 heartbeater.cc:464] Registering TS with master...
I20260806 03:59:39.230650 31702 heartbeater.cc:511] Master 127.29.237.190:38723 requested a full tablet report, sending...
I20260806 03:59:39.231485 30646 webserver.cc:460] Webserver started at http://127.29.237.130:39803/ using document root <none> and password file <none>
I20260806 03:59:39.232153 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 03:59:39.232398 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:59:39.232714 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:59:39.232950 31494 ts_manager.cc:194] Registered new tserver with Master: c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197)
I20260806 03:59:39.233873 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-1-root/instance:
uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
format_stamp: "Formatted at 2026-08-06 03:59:39 on dist-test-slave-lgr5"
I20260806 03:59:39.234745 31494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53434
I20260806 03:59:39.238511 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 03:59:39.241791 31720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.242504 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 03:59:39.242753 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-1-root
uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
format_stamp: "Formatted at 2026-08-06 03:59:39 on dist-test-slave-lgr5"
I20260806 03:59:39.242998 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:59:39.255781 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:59:39.256944 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:59:39.258339 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:59:39.260540 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:59:39.260728 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.260946 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:59:39.261085 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.298401 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.130:42307
I20260806 03:59:39.298506 31815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.130:42307 every 8 connection(s)
I20260806 03:59:39.302743 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:59:39.309612 31822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:39.310349 31823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:59:39.312300 31825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:59:39.312772 30646 server_base.cc:1034] running on GCE node
I20260806 03:59:39.313822 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:59:39.314021 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:59:39.314209 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988779314194 us; error 0 us; skew 500 ppm
I20260806 03:59:39.314653 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:59:39.316677 31816 heartbeater.cc:347] Connected to a master server at 127.29.237.190:38723
I20260806 03:59:39.317013 31816 heartbeater.cc:464] Registering TS with master...
I20260806 03:59:39.317521 30646 webserver.cc:460] Webserver started at http://127.29.237.131:46511/ using document root <none> and password file <none>
I20260806 03:59:39.317688 31816 heartbeater.cc:511] Master 127.29.237.190:38723 requested a full tablet report, sending...
I20260806 03:59:39.318037 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 03:59:39.318254 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:59:39.318490 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:59:39.319623 31494 ts_manager.cc:194] Registered new tserver with Master: 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:39.319890 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-2-root/instance:
uuid: "2166aaf7ebfc46f5ab80e355af820018"
format_stamp: "Formatted at 2026-08-06 03:59:39 on dist-test-slave-lgr5"
I20260806 03:59:39.321689 31494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53446
I20260806 03:59:39.324980 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 03:59:39.328214 31831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.328905 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 03:59:39.329141 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-2-root
uuid: "2166aaf7ebfc46f5ab80e355af820018"
format_stamp: "Formatted at 2026-08-06 03:59:39 on dist-test-slave-lgr5"
I20260806 03:59:39.329372 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988747097609-30646-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:59:39.359793 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:59:39.361475 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:59:39.363377 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:59:39.366305 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:59:39.366572 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.366864 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:59:39.367076 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:59:39.408434 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.131:35351
I20260806 03:59:39.408542 31919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.131:35351 every 8 connection(s)
I20260806 03:59:39.422894 31920 heartbeater.cc:347] Connected to a master server at 127.29.237.190:38723
I20260806 03:59:39.423251 31920 heartbeater.cc:464] Registering TS with master...
I20260806 03:59:39.423950 31920 heartbeater.cc:511] Master 127.29.237.190:38723 requested a full tablet report, sending...
I20260806 03:59:39.425750 31494 ts_manager.cc:194] Registered new tserver with Master: 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:39.425894 30646 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014568682s
I20260806 03:59:39.427326 31494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53458
I20260806 03:59:39.428215 30646 test_util.cc:268] Using random seed: -356169376
I20260806 03:59:39.452872 31494 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53474:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:59:39.455669 31494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:59:39.525285 31867 tablet_service.cc:1467] Processing CreateTablet for tablet 13da11470a074eb6be303c3ab41bd5cc (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:59:39.526710 31867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13da11470a074eb6be303c3ab41bd5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.529863 31866 tablet_service.cc:1467] Processing CreateTablet for tablet c4b0116d16a04a3f8012c44fc8c94be4 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:59:39.531251 31866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b0116d16a04a3f8012c44fc8c94be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.533257 31863 tablet_service.cc:1467] Processing CreateTablet for tablet d571ad6acd49444ab87adb4fec6d97ec (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:59:39.534564 31863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d571ad6acd49444ab87adb4fec6d97ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.537307 31861 tablet_service.cc:1467] Processing CreateTablet for tablet 54c76bab5b5d4876a48b6914768628aa (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:59:39.538630 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c76bab5b5d4876a48b6914768628aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.541373 31860 tablet_service.cc:1467] Processing CreateTablet for tablet 3b777807a4fe4cb9941725b6390218f3 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:59:39.542631 31860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b777807a4fe4cb9941725b6390218f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.543718 31754 tablet_service.cc:1467] Processing CreateTablet for tablet c4b0116d16a04a3f8012c44fc8c94be4 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:59:39.544193 31750 tablet_service.cc:1467] Processing CreateTablet for tablet 3b777807a4fe4cb9941725b6390218f3 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:59:39.545025 31754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b0116d16a04a3f8012c44fc8c94be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.546358 31749 tablet_service.cc:1467] Processing CreateTablet for tablet 0370dede28aa4bdf98ad4b7fb852e561 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:59:39.546815 31753 tablet_service.cc:1467] Processing CreateTablet for tablet 0cdbef037e374999a11b9e55e263ea6f (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:59:39.547694 31749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0370dede28aa4bdf98ad4b7fb852e561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.547515 31642 tablet_service.cc:1467] Processing CreateTablet for tablet c4b0116d16a04a3f8012c44fc8c94be4 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:59:39.548861 31642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b0116d16a04a3f8012c44fc8c94be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.549250 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 13da11470a074eb6be303c3ab41bd5cc (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:59:39.549707 31748 tablet_service.cc:1467] Processing CreateTablet for tablet 99ef48c15e7f49f0a55a11ffed3bcf0d (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:59:39.550064 31751 tablet_service.cc:1467] Processing CreateTablet for tablet 54c76bab5b5d4876a48b6914768628aa (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:59:39.550913 31639 tablet_service.cc:1467] Processing CreateTablet for tablet 0cdbef037e374999a11b9e55e263ea6f (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:59:39.542164 31755 tablet_service.cc:1467] Processing CreateTablet for tablet 13da11470a074eb6be303c3ab41bd5cc (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:59:39.551982 31747 tablet_service.cc:1467] Processing CreateTablet for tablet e0c56d39f4124b1397e875d0ebfc74a4 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:59:39.548820 31752 tablet_service.cc:1467] Processing CreateTablet for tablet d571ad6acd49444ab87adb4fec6d97ec (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:59:39.553222 31638 tablet_service.cc:1467] Processing CreateTablet for tablet d571ad6acd49444ab87adb4fec6d97ec (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:59:39.554157 31746 tablet_service.cc:1467] Processing CreateTablet for tablet bdc88796a69d4798bffcf0cbeb934ebc (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:59:39.555385 31636 tablet_service.cc:1467] Processing CreateTablet for tablet 54c76bab5b5d4876a48b6914768628aa (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:59:39.557519 31635 tablet_service.cc:1467] Processing CreateTablet for tablet 3b777807a4fe4cb9941725b6390218f3 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:59:39.559547 31857 tablet_service.cc:1467] Processing CreateTablet for tablet 0370dede28aa4bdf98ad4b7fb852e561 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:59:39.550946 31748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ef48c15e7f49f0a55a11ffed3bcf0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.560792 31857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0370dede28aa4bdf98ad4b7fb852e561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.560850 31752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d571ad6acd49444ab87adb4fec6d97ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.564445 31751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c76bab5b5d4876a48b6914768628aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.531896 31864 tablet_service.cc:1467] Processing CreateTablet for tablet 0cdbef037e374999a11b9e55e263ea6f (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:59:39.565400 31747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c56d39f4124b1397e875d0ebfc74a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.566094 31864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cdbef037e374999a11b9e55e263ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.568240 31639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cdbef037e374999a11b9e55e263ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.568873 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d571ad6acd49444ab87adb4fec6d97ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.569543 31635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b777807a4fe4cb9941725b6390218f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.570109 31753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cdbef037e374999a11b9e55e263ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.572433 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13da11470a074eb6be303c3ab41bd5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.576453 31746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc88796a69d4798bffcf0cbeb934ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.577860 31755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13da11470a074eb6be303c3ab41bd5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.579010 31750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b777807a4fe4cb9941725b6390218f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.578673 31634 tablet_service.cc:1467] Processing CreateTablet for tablet 0370dede28aa4bdf98ad4b7fb852e561 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:59:39.579989 31634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0370dede28aa4bdf98ad4b7fb852e561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.580497 31855 tablet_service.cc:1467] Processing CreateTablet for tablet 99ef48c15e7f49f0a55a11ffed3bcf0d (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:59:39.581278 31636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c76bab5b5d4876a48b6914768628aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.581842 31855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ef48c15e7f49f0a55a11ffed3bcf0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.584364 31853 tablet_service.cc:1467] Processing CreateTablet for tablet e0c56d39f4124b1397e875d0ebfc74a4 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:59:39.585632 31853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c56d39f4124b1397e875d0ebfc74a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.625077 31852 tablet_service.cc:1467] Processing CreateTablet for tablet bdc88796a69d4798bffcf0cbeb934ebc (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:59:39.626442 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc88796a69d4798bffcf0cbeb934ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.629788 31633 tablet_service.cc:1467] Processing CreateTablet for tablet 99ef48c15e7f49f0a55a11ffed3bcf0d (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:59:39.631196 31633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ef48c15e7f49f0a55a11ffed3bcf0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.637907 31632 tablet_service.cc:1467] Processing CreateTablet for tablet e0c56d39f4124b1397e875d0ebfc74a4 (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:59:39.639326 31632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c56d39f4124b1397e875d0ebfc74a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.647634 31943 tablet_bootstrap.cc:492] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:39.657691 31943 tablet_bootstrap.cc:654] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.701373 31631 tablet_service.cc:1467] Processing CreateTablet for tablet bdc88796a69d4798bffcf0cbeb934ebc (DEFAULT_TABLE table=test-workload [id=9c3e03cba7d84ca99041e2f29e227570]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:59:39.702777 31631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc88796a69d4798bffcf0cbeb934ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:59:39.712641 31945 tablet_bootstrap.cc:492] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
I20260806 03:59:39.723490 31945 tablet_bootstrap.cc:654] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.787940 31943 tablet_bootstrap.cc:492] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:39.788465 31943 ts_tablet_manager.cc:1397] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.141s user 0.028s sys 0.030s
I20260806 03:59:39.790395 31942 tablet_bootstrap.cc:492] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:39.797788 31943 raft_consensus.cc:348] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:39.798544 31943 raft_consensus.cc:374] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.798827 31943 raft_consensus.cc:729] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.799492 31943 consensus_queue.cc:260] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:39.802071 31945 tablet_bootstrap.cc:492] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:39.802505 31945 ts_tablet_manager.cc:1397] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.090s user 0.022s sys 0.042s
I20260806 03:59:39.804376 31942 tablet_bootstrap.cc:654] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.804944 31945 raft_consensus.cc:348] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.805658 31945 raft_consensus.cc:374] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.805934 31945 raft_consensus.cc:729] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.806597 31945 consensus_queue.cc:260] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.807523 31920 heartbeater.cc:503] Master 127.29.237.190:38723 was elected leader, sending a full tablet report...
I20260806 03:59:39.809033 31943 ts_tablet_manager.cc:1428] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260806 03:59:39.810026 31943 tablet_bootstrap.cc:492] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:39.817056 31943 tablet_bootstrap.cc:654] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.823729 31945 ts_tablet_manager.cc:1428] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.021s user 0.005s sys 0.017s
I20260806 03:59:39.823980 31816 heartbeater.cc:503] Master 127.29.237.190:38723 was elected leader, sending a full tablet report...
I20260806 03:59:39.824880 31945 tablet_bootstrap.cc:492] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
I20260806 03:59:39.831630 31945 tablet_bootstrap.cc:654] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.836699 31943 tablet_bootstrap.cc:492] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:39.837158 31943 ts_tablet_manager.cc:1397] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20260806 03:59:39.837692 31945 tablet_bootstrap.cc:492] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:39.838238 31942 tablet_bootstrap.cc:492] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:39.838270 31945 ts_tablet_manager.cc:1397] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.002s
I20260806 03:59:39.838809 31942 ts_tablet_manager.cc:1397] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.005s
I20260806 03:59:39.839821 31943 raft_consensus.cc:348] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.840674 31943 raft_consensus.cc:374] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.840961 31943 raft_consensus.cc:729] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.840986 31945 raft_consensus.cc:348] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.841761 31945 raft_consensus.cc:374] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.842177 31945 raft_consensus.cc:729] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.841794 31942 raft_consensus.cc:348] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.841850 31943 consensus_queue.cc:260] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.842857 31942 raft_consensus.cc:374] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.843309 31942 raft_consensus.cc:729] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.842836 31945 consensus_queue.cc:260] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.844360 31943 ts_tablet_manager.cc:1428] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 03:59:39.844373 31942 consensus_queue.cc:260] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.845201 31945 ts_tablet_manager.cc:1428] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260806 03:59:39.845274 31943 tablet_bootstrap.cc:492] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:39.846184 31945 tablet_bootstrap.cc:492] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
I20260806 03:59:39.851397 31943 tablet_bootstrap.cc:654] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.852218 31945 tablet_bootstrap.cc:654] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.854038 31942 ts_tablet_manager.cc:1428] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.015s user 0.005s sys 0.007s
I20260806 03:59:39.854758 31702 heartbeater.cc:503] Master 127.29.237.190:38723 was elected leader, sending a full tablet report...
I20260806 03:59:39.860661 31942 tablet_bootstrap.cc:492] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:39.863996 31943 tablet_bootstrap.cc:492] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:39.864511 31943 ts_tablet_manager.cc:1397] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.014s
I20260806 03:59:39.866451 31942 tablet_bootstrap.cc:654] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.867211 31943 raft_consensus.cc:348] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.867996 31943 raft_consensus.cc:374] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.868549 31943 raft_consensus.cc:729] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.869472 31945 tablet_bootstrap.cc:492] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:39.869823 31943 consensus_queue.cc:260] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.869902 31945 ts_tablet_manager.cc:1397] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.000s
I20260806 03:59:39.872699 31943 ts_tablet_manager.cc:1428] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:59:39.873608 31943 tablet_bootstrap.cc:492] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:39.874712 31945 raft_consensus.cc:348] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.875437 31945 raft_consensus.cc:374] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.875711 31945 raft_consensus.cc:729] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.877317 31945 consensus_queue.cc:260] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.879671 31945 ts_tablet_manager.cc:1428] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 03:59:39.879779 31943 tablet_bootstrap.cc:654] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.880630 31945 tablet_bootstrap.cc:492] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
I20260806 03:59:39.886835 31945 tablet_bootstrap.cc:654] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.896471 31945 tablet_bootstrap.cc:492] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:39.896905 31945 ts_tablet_manager.cc:1397] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260806 03:59:39.897109 31943 tablet_bootstrap.cc:492] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:39.897563 31943 ts_tablet_manager.cc:1397] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.004s
I20260806 03:59:39.899411 31945 raft_consensus.cc:348] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.900233 31945 raft_consensus.cc:374] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.900002 31943 raft_consensus.cc:348] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.900537 31945 raft_consensus.cc:729] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.900781 31943 raft_consensus.cc:374] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.901067 31943 raft_consensus.cc:729] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.901394 31945 consensus_queue.cc:260] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.901830 31943 consensus_queue.cc:260] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.903909 31943 ts_tablet_manager.cc:1428] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 03:59:39.904888 31943 tablet_bootstrap.cc:492] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:39.905710 31945 ts_tablet_manager.cc:1428] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260806 03:59:39.906553 31945 tablet_bootstrap.cc:492] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
I20260806 03:59:39.910805 31943 tablet_bootstrap.cc:654] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.913869 31945 tablet_bootstrap.cc:654] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.926215 31945 tablet_bootstrap.cc:492] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:39.926735 31945 ts_tablet_manager.cc:1397] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.006s
I20260806 03:59:39.929898 31945 raft_consensus.cc:348] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.930853 31945 raft_consensus.cc:374] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.931205 31945 raft_consensus.cc:729] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.931908 31945 consensus_queue.cc:260] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.932559 31942 tablet_bootstrap.cc:492] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:39.933008 31942 ts_tablet_manager.cc:1397] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.073s user 0.015s sys 0.045s
I20260806 03:59:39.933635 31943 tablet_bootstrap.cc:492] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:39.934141 31943 ts_tablet_manager.cc:1397] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.009s
I20260806 03:59:39.934478 31945 ts_tablet_manager.cc:1428] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260806 03:59:39.935427 31945 tablet_bootstrap.cc:492] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
I20260806 03:59:39.935670 31942 raft_consensus.cc:348] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.936410 31942 raft_consensus.cc:374] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.936733 31942 raft_consensus.cc:729] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.936666 31943 raft_consensus.cc:348] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.937434 31943 raft_consensus.cc:374] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.937729 31943 raft_consensus.cc:729] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.937673 31942 consensus_queue.cc:260] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.938637 31943 consensus_queue.cc:260] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.941754 31945 tablet_bootstrap.cc:654] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.942049 31942 ts_tablet_manager.cc:1428] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 03:59:39.942917 31942 tablet_bootstrap.cc:492] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:39.949656 31942 tablet_bootstrap.cc:654] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.950260 31943 ts_tablet_manager.cc:1428] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.016s user 0.001s sys 0.004s
I20260806 03:59:39.952993 31945 tablet_bootstrap.cc:492] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:39.953444 31945 ts_tablet_manager.cc:1397] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260806 03:59:39.953588 31943 tablet_bootstrap.cc:492] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:39.956065 31945 raft_consensus.cc:348] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.956928 31945 raft_consensus.cc:374] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.957221 31945 raft_consensus.cc:729] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.957949 31945 consensus_queue.cc:260] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.959419 31943 tablet_bootstrap.cc:654] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.962471 31942 tablet_bootstrap.cc:492] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:39.962514 31945 ts_tablet_manager.cc:1428] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 03:59:39.963054 31942 ts_tablet_manager.cc:1397] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20260806 03:59:39.963639 31945 tablet_bootstrap.cc:492] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
I20260806 03:59:39.965782 31942 raft_consensus.cc:348] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.966562 31942 raft_consensus.cc:374] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.966863 31942 raft_consensus.cc:729] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.967533 31942 consensus_queue.cc:260] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.969925 31945 tablet_bootstrap.cc:654] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.971159 31942 ts_tablet_manager.cc:1428] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 03:59:39.972118 31942 tablet_bootstrap.cc:492] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:39.978284 31942 tablet_bootstrap.cc:654] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.983724 31942 tablet_bootstrap.cc:492] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:39.984201 31942 ts_tablet_manager.cc:1397] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260806 03:59:39.986203 31943 tablet_bootstrap.cc:492] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:39.986660 31943 ts_tablet_manager.cc:1397] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.013s
I20260806 03:59:39.986555 31942 raft_consensus.cc:348] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.987164 31942 raft_consensus.cc:374] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.987205 31945 tablet_bootstrap.cc:492] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:39.987423 31942 raft_consensus.cc:729] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.987691 31945 ts_tablet_manager.cc:1397] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.001s
I20260806 03:59:39.988008 31942 consensus_queue.cc:260] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.989281 31943 raft_consensus.cc:348] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.989981 31942 ts_tablet_manager.cc:1428] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260806 03:59:39.990008 31943 raft_consensus.cc:374] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.990486 31943 raft_consensus.cc:729] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.990175 31945 raft_consensus.cc:348] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.990899 31945 raft_consensus.cc:374] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:39.991039 31942 tablet_bootstrap.cc:492] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:39.991178 31945 raft_consensus.cc:729] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:39.991163 31943 consensus_queue.cc:260] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:39.993489 31943 ts_tablet_manager.cc:1428] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 03:59:39.994382 31943 tablet_bootstrap.cc:492] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:39.997085 31942 tablet_bootstrap.cc:654] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.000126 31943 tablet_bootstrap.cc:654] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:39.992064 31945 consensus_queue.cc:260] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.006349 31945 ts_tablet_manager.cc:1428] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s
I20260806 03:59:40.008349 31948 raft_consensus.cc:482] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.008864 31948 raft_consensus.cc:504] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.010751 31942 tablet_bootstrap.cc:492] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:40.011199 31942 ts_tablet_manager.cc:1397] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20260806 03:59:40.011420 31948 leader_election.cc:288] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.013877 31942 raft_consensus.cc:348] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.014642 31942 raft_consensus.cc:374] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.014871 31943 tablet_bootstrap.cc:492] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:40.014930 31942 raft_consensus.cc:729] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.015596 31943 ts_tablet_manager.cc:1397] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.009s
I20260806 03:59:40.015852 31942 consensus_queue.cc:260] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.018342 31943 raft_consensus.cc:348] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.019106 31943 raft_consensus.cc:374] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.019430 31943 raft_consensus.cc:729] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.020221 31943 consensus_queue.cc:260] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.021076 31942 ts_tablet_manager.cc:1428] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260806 03:59:40.022002 31942 tablet_bootstrap.cc:492] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:40.022501 31943 ts_tablet_manager.cc:1428] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 03:59:40.023430 31943 tablet_bootstrap.cc:492] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:40.027714 31942 tablet_bootstrap.cc:654] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.029196 31943 tablet_bootstrap.cc:654] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.007874 31945 tablet_bootstrap.cc:492] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
I20260806 03:59:40.041796 31945 tablet_bootstrap.cc:654] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.042622 31943 tablet_bootstrap.cc:492] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:40.043088 31943 ts_tablet_manager.cc:1397] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.001s
I20260806 03:59:40.044109 31942 tablet_bootstrap.cc:492] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:40.044566 31942 ts_tablet_manager.cc:1397] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.012s
I20260806 03:59:40.045632 31943 raft_consensus.cc:348] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.046312 31943 raft_consensus.cc:374] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.046590 31943 raft_consensus.cc:729] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.047148 31943 consensus_queue.cc:260] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
W20260806 03:59:40.047688 31574 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:59:40.049366 31943 ts_tablet_manager.cc:1428] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260806 03:59:40.049198 31942 raft_consensus.cc:348] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.050037 31942 raft_consensus.cc:374] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.050247 31943 tablet_bootstrap.cc:492] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:40.050355 31942 raft_consensus.cc:729] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.051178 31942 consensus_queue.cc:260] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.057301 31942 ts_tablet_manager.cc:1428] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.012s user 0.000s sys 0.006s
I20260806 03:59:40.058225 31943 tablet_bootstrap.cc:654] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.058353 31942 tablet_bootstrap.cc:492] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:40.061439 31957 raft_consensus.cc:482] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.061906 31957 raft_consensus.cc:504] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.064436 31957 leader_election.cc:288] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.068019 31771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.068866 31771 raft_consensus.cc:2454] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.069743 31942 tablet_bootstrap.cc:654] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.072943 31833 leader_election.cc:302] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:40.073794 31948 raft_consensus.cc:2781] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.074139 31948 raft_consensus.cc:482] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.074458 31948 raft_consensus.cc:3037] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.083009 31948 raft_consensus.cc:504] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.086499 31771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.087103 31771 raft_consensus.cc:3037] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.090999 31945 tablet_bootstrap.cc:492] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:40.091493 31945 ts_tablet_manager.cc:1397] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.005s
I20260806 03:59:40.093604 31771 raft_consensus.cc:2454] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 1.
I20260806 03:59:40.094103 31945 raft_consensus.cc:348] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.094759 31833 leader_election.cc:302] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:40.098562 31948 leader_election.cc:288] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.100564 31948 raft_consensus.cc:2781] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.101006 31948 raft_consensus.cc:686] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:40.101818 31948 consensus_queue.cc:237] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.110416 31660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.111888 31658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.113116 31950 raft_consensus.cc:482] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.113574 31950 raft_consensus.cc:504] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.094759 31945 raft_consensus.cc:374] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.114478 31945 raft_consensus.cc:729] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.115226 31945 consensus_queue.cc:260] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.119479 31950 leader_election.cc:288] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.122089 31943 tablet_bootstrap.cc:492] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:40.122504 31942 tablet_bootstrap.cc:492] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:40.122561 31943 ts_tablet_manager.cc:1397] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.072s user 0.042s sys 0.023s
I20260806 03:59:40.123070 31942 ts_tablet_manager.cc:1397] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.035s
I20260806 03:59:40.125257 31943 raft_consensus.cc:348] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.126014 31943 raft_consensus.cc:374] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.126377 31943 raft_consensus.cc:729] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.127128 31943 consensus_queue.cc:260] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
W20260806 03:59:40.128933 31832 leader_election.cc:341] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:59:40.129945 31832 leader_election.cc:341] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:40.125768 31942 raft_consensus.cc:348] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.130929 31942 raft_consensus.cc:374] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.131237 31942 raft_consensus.cc:729] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.132050 31942 consensus_queue.cc:260] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.132470 31998 raft_consensus.cc:482] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.132977 31998 raft_consensus.cc:504] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.134367 31942 ts_tablet_manager.cc:1428] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20260806 03:59:40.135372 31942 tablet_bootstrap.cc:492] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:40.135960 31658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571ad6acd49444ab87adb4fec6d97ec" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.136662 31658 raft_consensus.cc:2454] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.137853 31832 leader_election.cc:302] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.139271 31943 ts_tablet_manager.cc:1428] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.016s user 0.006s sys 0.003s
I20260806 03:59:40.140396 31943 tablet_bootstrap.cc:492] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018: Bootstrap starting.
I20260806 03:59:40.146777 31999 raft_consensus.cc:2781] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.147369 31999 raft_consensus.cc:482] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.148897 31999 raft_consensus.cc:3037] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.148450 31998 leader_election.cc:288] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.151531 31943 tablet_bootstrap.cc:654] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.158315 31998 raft_consensus.cc:482] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.158900 31998 raft_consensus.cc:504] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.163419 31998 leader_election.cc:288] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.170324 31771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.178252 31945 ts_tablet_manager.cc:1428] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.086s user 0.025s sys 0.012s
I20260806 03:59:40.179371 31945 tablet_bootstrap.cc:492] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
W20260806 03:59:40.182837 31833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:42307, user_credentials={real_user=slave}} blocked reactor thread for 88163.2us
I20260806 03:59:40.184888 31770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571ad6acd49444ab87adb4fec6d97ec" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.185644 31770 raft_consensus.cc:2454] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.188148 31770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc88796a69d4798bffcf0cbeb934ebc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.178654 31771 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 0.
I20260806 03:59:40.189970 31943 tablet_bootstrap.cc:492] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018: No bootstrap required, opened a new log
I20260806 03:59:40.190508 31943 ts_tablet_manager.cc:1397] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.012s
I20260806 03:59:40.191186 31993 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.188787 31770 raft_consensus.cc:2454] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.193078 31833 leader_election.cc:302] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:40.196043 31998 raft_consensus.cc:2781] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.196421 31998 raft_consensus.cc:482] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.191663 31993 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.194667 31943 raft_consensus.cc:348] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.196743 31998 raft_consensus.cc:3037] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.197618 31943 raft_consensus.cc:374] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.198670 31943 raft_consensus.cc:729] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.199311 31943 consensus_queue.cc:260] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.205904 31945 tablet_bootstrap.cc:654] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.205559 31998 raft_consensus.cc:504] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.206840 31999 raft_consensus.cc:504] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.209640 31998 leader_election.cc:288] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.210430 31770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571ad6acd49444ab87adb4fec6d97ec" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.210798 31942 tablet_bootstrap.cc:654] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.211066 31770 raft_consensus.cc:3037] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.221721 31770 raft_consensus.cc:2454] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 1.
I20260806 03:59:40.213830 31999 leader_election.cc:288] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.232537 31957 raft_consensus.cc:482] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.232858 31993 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.233096 31957 raft_consensus.cc:504] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.233544 31994 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.234067 31994 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.238453 31994 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.242938 31943 ts_tablet_manager.cc:1428] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018: Time spent starting tablet: real 0.052s user 0.006s sys 0.022s
I20260806 03:59:40.247931 31957 leader_election.cc:288] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.273758 31658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc88796a69d4798bffcf0cbeb934ebc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.274531 31658 raft_consensus.cc:2454] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.275761 31657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571ad6acd49444ab87adb4fec6d97ec" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.276468 31657 raft_consensus.cc:3037] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.280593 32003 raft_consensus.cc:482] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.274438 31660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc88796a69d4798bffcf0cbeb934ebc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.281133 32003 raft_consensus.cc:504] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.281595 31660 raft_consensus.cc:3037] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.285933 31657 raft_consensus.cc:2454] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 1.
I20260806 03:59:40.287707 31660 raft_consensus.cc:2454] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 1.
I20260806 03:59:40.288987 32003 leader_election.cc:288] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
W20260806 03:59:40.312417 31817 tablet.cc:2369] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:59:40.487286 31942 tablet_bootstrap.cc:492] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:40.487820 31942 ts_tablet_manager.cc:1397] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.353s user 0.018s sys 0.023s
I20260806 03:59:40.491446 31942 raft_consensus.cc:348] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.492393 31942 raft_consensus.cc:374] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.492687 31942 raft_consensus.cc:729] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.493309 31942 consensus_queue.cc:260] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.495461 31942 ts_tablet_manager.cc:1428] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 03:59:40.496266 31942 tablet_bootstrap.cc:492] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:40.501134 31942 tablet_bootstrap.cc:654] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.509479 31942 tablet_bootstrap.cc:492] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:40.509856 31942 ts_tablet_manager.cc:1397] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260806 03:59:40.512341 31942 raft_consensus.cc:348] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.513015 31942 raft_consensus.cc:374] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.513252 31942 raft_consensus.cc:729] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.513978 31942 consensus_queue.cc:260] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.516152 32011 raft_consensus.cc:482] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.516517 31942 ts_tablet_manager.cc:1428] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 03:59:40.516698 32011 raft_consensus.cc:504] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.517405 31942 tablet_bootstrap.cc:492] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79: Bootstrap starting.
I20260806 03:59:40.519167 32011 leader_election.cc:288] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.521205 32011 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.521724 32011 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.521860 31878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.522567 31878 raft_consensus.cc:2454] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 0.
I20260806 03:59:40.523319 32011 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.524050 31607 leader_election.cc:302] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.524828 31771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc88796a69d4798bffcf0cbeb934ebc" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.525359 32011 raft_consensus.cc:2781] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.525467 31771 raft_consensus.cc:2454] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 0.
I20260806 03:59:40.525612 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc88796a69d4798bffcf0cbeb934ebc" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.526585 31880 raft_consensus.cc:2379] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c7dd78b16d8842e88ec02c12a3a6ba79 in current term 1: Already voted for candidate 2166aaf7ebfc46f5ab80e355af820018 in this term.
I20260806 03:59:40.526803 31942 tablet_bootstrap.cc:654] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.527117 31606 leader_election.cc:302] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.527551 31771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.527992 31606 leader_election.cc:302] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.528158 31771 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 0.
I20260806 03:59:40.521245 32009 raft_consensus.cc:482] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.525677 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.525743 32011 raft_consensus.cc:482] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.529770 32011 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.530488 31878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.531082 31878 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 0.
I20260806 03:59:40.532307 31607 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.537237 31770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc88796a69d4798bffcf0cbeb934ebc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.537528 31945 tablet_bootstrap.cc:492] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:40.538017 31945 ts_tablet_manager.cc:1397] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.359s user 0.008s sys 0.033s
I20260806 03:59:40.538467 32012 raft_consensus.cc:2781] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.538720 31994 raft_consensus.cc:482] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.538878 32012 raft_consensus.cc:482] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.536984 32011 raft_consensus.cc:504] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.539217 31994 raft_consensus.cc:504] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.541010 32011 leader_election.cc:288] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.540737 31945 raft_consensus.cc:348] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.540678 31770 raft_consensus.cc:3037] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.541433 31945 raft_consensus.cc:374] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.542165 31945 raft_consensus.cc:729] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.542779 31945 consensus_queue.cc:260] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.543150 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.543727 31767 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.541514 31994 leader_election.cc:288] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.544749 31994 raft_consensus.cc:482] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.545154 31994 raft_consensus.cc:504] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.547499 31994 leader_election.cc:288] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.550637 31767 raft_consensus.cc:2454] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 1.
I20260806 03:59:40.528993 32009 raft_consensus.cc:504] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.552039 31770 raft_consensus.cc:2454] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 1.
I20260806 03:59:40.552934 32009 leader_election.cc:288] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.558094 32009 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.558423 32009 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.558750 32009 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.559090 32003 raft_consensus.cc:482] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.559553 32003 raft_consensus.cc:504] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.561403 32003 leader_election.cc:288] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.561861 32003 raft_consensus.cc:482] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.562238 32003 raft_consensus.cc:504] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.566182 32015 raft_consensus.cc:482] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.566619 32015 raft_consensus.cc:504] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.567675 31878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.568291 31878 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.553552 31945 ts_tablet_manager.cc:1428] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.015s user 0.004s sys 0.007s
I20260806 03:59:40.553818 32011 raft_consensus.cc:2741] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260806 03:59:40.572242 31945 tablet_bootstrap.cc:492] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815: Bootstrap starting.
W20260806 03:59:40.553464 31606 leader_election.cc:341] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:40.573355 31606 leader_election.cc:302] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.574086 32011 raft_consensus.cc:2781] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.574540 32011 raft_consensus.cc:686] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:40.575599 32011 consensus_queue.cc:237] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.539219 32012 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.563966 32003 leader_election.cc:288] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.558702 31832 leader_election.cc:302] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.588512 32015 leader_election.cc:288] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.580030 31770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76bab5b5d4876a48b6914768628aa" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.593866 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76bab5b5d4876a48b6914768628aa" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.594565 31880 raft_consensus.cc:2454] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 0.
I20260806 03:59:40.595341 31660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.596141 31657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.596786 32003 raft_consensus.cc:2781] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.600097 31945 tablet_bootstrap.cc:654] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815: Neither blocks nor log segments found. Creating new log.
I20260806 03:59:40.596828 31770 raft_consensus.cc:2454] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 0.
I20260806 03:59:40.601219 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.598683 31832 leader_election.cc:302] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.601799 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.602527 31658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.604297 32003 raft_consensus.cc:686] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:40.605803 31658 raft_consensus.cc:2454] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 0.
I20260806 03:59:40.593392 32009 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.604027 31606 leader_election.cc:302] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.608593 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76bab5b5d4876a48b6914768628aa" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.610122 31656 raft_consensus.cc:2454] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 0.
I20260806 03:59:40.606560 32014 raft_consensus.cc:482] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.611027 32014 raft_consensus.cc:504] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.609508 31999 raft_consensus.cc:2781] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.612815 31999 raft_consensus.cc:686] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:40.613806 32014 leader_election.cc:288] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.613616 31999 consensus_queue.cc:237] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.614512 32003 consensus_queue.cc:237] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.614938 31770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
W20260806 03:59:40.616240 31606 leader_election.cc:341] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:40.616871 31658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.618072 31957 raft_consensus.cc:2781] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.623116 31957 raft_consensus.cc:482] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.623493 31957 raft_consensus.cc:3037] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.607013 31880 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 0.
I20260806 03:59:40.611820 31877 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 0.
I20260806 03:59:40.630002 31942 tablet_bootstrap.cc:492] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79: No bootstrap required, opened a new log
I20260806 03:59:40.630481 31942 ts_tablet_manager.cc:1397] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent bootstrapping tablet: real 0.113s user 0.024s sys 0.025s
I20260806 03:59:40.620433 31654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.633152 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.628206 31993 raft_consensus.cc:482] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.634096 31942 raft_consensus.cc:348] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.636440 31770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.641183 31942 raft_consensus.cc:374] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.642253 31942 raft_consensus.cc:729] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.642890 31942 consensus_queue.cc:260] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.641188 31657 raft_consensus.cc:2379] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in current term 1: Already voted for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in this term.
I20260806 03:59:40.651680 31942 ts_tablet_manager.cc:1428] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260806 03:59:40.645545 31770 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.645996 31656 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.642836 32012 raft_consensus.cc:504] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.659149 31722 leader_election.cc:302] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.660063 31722 leader_election.cc:302] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
W20260806 03:59:40.660924 31722 leader_election.cc:341] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:59:40.661499 32015 raft_consensus.cc:2781] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.661847 32015 raft_consensus.cc:482] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.662150 32015 raft_consensus.cc:3037] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.662848 31945 tablet_bootstrap.cc:492] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815: No bootstrap required, opened a new log
I20260806 03:59:40.663481 31945 ts_tablet_manager.cc:1397] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent bootstrapping tablet: real 0.091s user 0.030s sys 0.010s
I20260806 03:59:40.666854 31950 raft_consensus.cc:2781] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.667258 31950 raft_consensus.cc:482] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.665808 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.667709 31950 raft_consensus.cc:3037] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.666738 31874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.654230 31658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.644487 31993 raft_consensus.cc:504] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.669661 31658 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.668715 32014 raft_consensus.cc:482] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:40.666631 31653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.667917 31767 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.671564 32014 raft_consensus.cc:504] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.672374 31653 raft_consensus.cc:2454] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.673239 32009 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.673863 31660 raft_consensus.cc:2379] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in current term 1: Already voted for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in this term.
I20260806 03:59:40.674152 31770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.673869 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.657517 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76bab5b5d4876a48b6914768628aa" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.674660 31882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.674985 31880 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.677168 31957 raft_consensus.cc:504] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.678292 31771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.679240 31877 raft_consensus.cc:2454] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 0.
I20260806 03:59:40.674621 32014 leader_election.cc:288] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.676333 31873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.680591 31873 raft_consensus.cc:2454] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 0.
I20260806 03:59:40.675635 31833 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters: c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.672333 32012 leader_election.cc:288] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.683614 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.683552 31724 leader_election.cc:302] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:40.672823 31878 raft_consensus.cc:2454] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 1.
I20260806 03:59:40.681818 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.683465 31832 leader_election.cc:302] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.688532 31994 raft_consensus.cc:2781] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.688863 31994 raft_consensus.cc:482] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.689183 31994 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.690230 31957 leader_election.cc:288] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.687950 31945 raft_consensus.cc:348] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.691411 31945 raft_consensus.cc:374] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:59:40.691741 31945 raft_consensus.cc:729] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Initialized, Role: FOLLOWER
I20260806 03:59:40.685688 31999 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.692392 31945 consensus_queue.cc:260] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.688984 31832 leader_election.cc:302] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.678917 31874 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 0.
I20260806 03:59:40.687419 31724 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters: c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.693519 31999 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.695025 32003 raft_consensus.cc:2781] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.702752 31764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.704504 31880 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 1.
I20260806 03:59:40.706879 31999 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.693650 31882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018" is_pre_election: true
I20260806 03:59:40.710856 31882 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 0.
I20260806 03:59:40.711838 32015 raft_consensus.cc:504] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.711932 31494 catalog_manager.cc:5495] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:40.713617 32015 leader_election.cc:288] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.708457 31945 ts_tablet_manager.cc:1428] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815: Time spent starting tablet: real 0.045s user 0.009s sys 0.000s
I20260806 03:59:40.707152 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76bab5b5d4876a48b6914768628aa" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.694231 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.709681 31607 leader_election.cc:302] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters: 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:40.709758 31874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76bab5b5d4876a48b6914768628aa" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.716346 32023 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.712904 31771 raft_consensus.cc:2379] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2166aaf7ebfc46f5ab80e355af820018 in current term 1: Already voted for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in this term.
I20260806 03:59:40.708256 31948 raft_consensus.cc:2781] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.716732 31874 raft_consensus.cc:3037] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.706154 31993 leader_election.cc:288] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.729506 31770 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 0.
I20260806 03:59:40.731585 31607 leader_election.cc:302] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.717612 31764 raft_consensus.cc:2379] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c7dd78b16d8842e88ec02c12a3a6ba79 in current term 1: Already voted for candidate 2166aaf7ebfc46f5ab80e355af820018 in this term.
I20260806 03:59:40.737104 31950 raft_consensus.cc:504] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.738346 31763 raft_consensus.cc:2379] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7dd78b16d8842e88ec02c12a3a6ba79 in current term 1: Already voted for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in this term.
I20260806 03:59:40.738369 31606 leader_election.cc:302] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7dd78b16d8842e88ec02c12a3a6ba79; no voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:40.706043 31833 leader_election.cc:302] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:40.740262 31993 raft_consensus.cc:2781] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.740635 31993 raft_consensus.cc:482] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.740967 31993 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.722626 31999 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.742079 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:40.724395 31994 raft_consensus.cc:504] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.732281 31957 raft_consensus.cc:2781] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.732939 31660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.700495 31765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.728500 31948 raft_consensus.cc:482] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.733654 32023 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.735976 32003 raft_consensus.cc:482] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.740265 32012 raft_consensus.cc:3037] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:59:40.755334 31724 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:35351, user_credentials={real_user=slave}} blocked reactor thread for 67987.4us
I20260806 03:59:40.755510 31763 raft_consensus.cc:2379] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2166aaf7ebfc46f5ab80e355af820018 in current term 1: Already voted for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in this term.
I20260806 03:59:40.750696 31957 raft_consensus.cc:482] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.755944 31957 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.752955 31877 raft_consensus.cc:2379] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7dd78b16d8842e88ec02c12a3a6ba79 in current term 1: Already voted for candidate 2166aaf7ebfc46f5ab80e355af820018 in this term.
I20260806 03:59:40.751889 31948 raft_consensus.cc:3037] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.752811 32014 raft_consensus.cc:2781] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.758288 31950 leader_election.cc:288] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.752584 32003 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.758807 31999 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.759881 32014 raft_consensus.cc:686] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:40.760326 31999 consensus_queue.cc:1035] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:40.751595 32023 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.752992 31660 raft_consensus.cc:3037] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.760807 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.764649 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76bab5b5d4876a48b6914768628aa" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.765034 31765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.765362 31656 raft_consensus.cc:2379] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in current term 1: Already voted for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in this term.
I20260806 03:59:40.766585 31882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76bab5b5d4876a48b6914768628aa" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
W20260806 03:59:40.768298 31832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:37197, user_credentials={real_user=slave}} blocked reactor thread for 52884.5us
I20260806 03:59:40.770560 31874 raft_consensus.cc:2454] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 1.
I20260806 03:59:40.770869 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:40.770756 31658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.772774 31658 raft_consensus.cc:2379] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2166aaf7ebfc46f5ab80e355af820018 in current term 1: Already voted for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in this term.
I20260806 03:59:40.776391 31833 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018; no voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.776671 31993 raft_consensus.cc:504] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.779158 31999 raft_consensus.cc:2726] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.780398 31658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.780961 31658 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.760747 32014 consensus_queue.cc:237] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.779752 31833 leader_election.cc:302] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018; no voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.785588 31999 raft_consensus.cc:2726] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.786692 31994 leader_election.cc:288] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.790630 31607 leader_election.cc:302] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters: 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:40.790766 31724 leader_election.cc:302] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters: c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.791437 32009 raft_consensus.cc:2781] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.796296 32009 raft_consensus.cc:686] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:40.796531 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.797197 32009 consensus_queue.cc:237] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.803650 32012 raft_consensus.cc:2726] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.807255 31957 raft_consensus.cc:504] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.807736 31999 consensus_queue.cc:1035] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:59:40.815136 32023 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.816246 31767 raft_consensus.cc:2379] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7dd78b16d8842e88ec02c12a3a6ba79 in current term 1: Already voted for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in this term.
I20260806 03:59:40.817942 31994 raft_consensus.cc:2781] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:59:40.820729 31994 raft_consensus.cc:482] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:40.821086 31994 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.817637 31948 raft_consensus.cc:504] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.820158 32023 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.794678 31658 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 1.
I20260806 03:59:40.825074 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.827840 31724 leader_election.cc:302] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815; no voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.829118 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.831104 31606 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7dd78b16d8842e88ec02c12a3a6ba79; no voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:40.818168 31993 leader_election.cc:288] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.834404 31832 leader_election.cc:302] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.834430 31880 raft_consensus.cc:2379] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in current term 1: Already voted for candidate 2166aaf7ebfc46f5ab80e355af820018 in this term.
I20260806 03:59:40.835677 31656 raft_consensus.cc:2379] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in current term 1: Already voted for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in this term.
I20260806 03:59:40.836473 32023 raft_consensus.cc:2726] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.829878 31660 raft_consensus.cc:2454] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 1.
I20260806 03:59:40.839897 31948 leader_election.cc:288] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.838902 31994 raft_consensus.cc:504] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.841434 31874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.854334 31874 raft_consensus.cc:2379] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in current term 1: Already voted for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in this term.
I20260806 03:59:40.854419 31763 raft_consensus.cc:2379] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2166aaf7ebfc46f5ab80e355af820018 in current term 1: Already voted for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in this term.
I20260806 03:59:40.854666 31722 leader_election.cc:302] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:40.855470 32023 raft_consensus.cc:2781] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.855912 32023 raft_consensus.cc:686] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 LEADER]: Becoming Leader. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:40.856935 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.857533 31763 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:59:40.857997 31724 leader_election.cc:302] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815; no voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.858323 31660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.858772 31950 raft_consensus.cc:2726] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.859184 31660 raft_consensus.cc:2379] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in current term 1: Already voted for candidate 2166aaf7ebfc46f5ab80e355af820018 in this term.
I20260806 03:59:40.837564 32009 raft_consensus.cc:2726] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.860172 31957 leader_election.cc:288] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.839468 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.852596 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.833801 32003 raft_consensus.cc:504] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.839818 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.837906 31993 raft_consensus.cc:2781] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.863200 31874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:40.861843 31994 leader_election.cc:288] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:40.864797 31877 raft_consensus.cc:2379] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in current term 1: Already voted for candidate 2166aaf7ebfc46f5ab80e355af820018 in this term.
I20260806 03:59:40.856701 32023 consensus_queue.cc:237] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.866884 31660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.867123 31724 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815; no voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.867616 31660 raft_consensus.cc:2379] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2166aaf7ebfc46f5ab80e355af820018 in current term 1: Already voted for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in this term.
I20260806 03:59:40.867555 31993 raft_consensus.cc:686] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:40.868237 31994 raft_consensus.cc:2726] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.839816 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:40.869956 31656 raft_consensus.cc:2379] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2166aaf7ebfc46f5ab80e355af820018 in current term 1: Already voted for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in this term.
I20260806 03:59:40.868489 31993 consensus_queue.cc:237] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:40.877722 31880 raft_consensus.cc:2379] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7dd78b16d8842e88ec02c12a3a6ba79 in current term 1: Already voted for candidate 2166aaf7ebfc46f5ab80e355af820018 in this term.
I20260806 03:59:40.879066 31832 leader_election.cc:302] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018; no voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.879947 31948 raft_consensus.cc:2726] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.883276 32003 leader_election.cc:288] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:40.884402 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:40.885119 31763 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 1.
I20260806 03:59:40.885950 31833 leader_election.cc:302] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018; no voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.888154 31606 leader_election.cc:302] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters: 2166aaf7ebfc46f5ab80e355af820018
I20260806 03:59:40.889068 32009 raft_consensus.cc:2781] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:59:40.889555 32009 raft_consensus.cc:686] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:40.889643 32003 raft_consensus.cc:2726] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.891359 32009 consensus_queue.cc:237] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:40.902089 31724 leader_election.cc:302] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815; no voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:40.902887 31994 raft_consensus.cc:2726] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:59:40.912981 31493 catalog_manager.cc:5495] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:40.923938 31488 catalog_manager.cc:5495] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:40.925928 31488 catalog_manager.cc:5495] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:40.928021 31488 catalog_manager.cc:5495] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:40.939960 31494 catalog_manager.cc:5495] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:40.949515 31493 catalog_manager.cc:5495] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:40.951306 31493 catalog_manager.cc:5495] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:40.952585 31493 catalog_manager.cc:5495] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:40.989267 32009 consensus_queue.cc:1035] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.000653 32009 consensus_queue.cc:1035] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:59:41.049813 31574 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:59:41.220970 32009 consensus_queue.cc:1035] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.224710 31999 consensus_queue.cc:1035] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.226102 31999 consensus_queue.cc:1035] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.227380 31999 consensus_queue.cc:1035] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.229491 31999 consensus_queue.cc:1035] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.239842 32014 consensus_queue.cc:1035] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 03:59:41.261107 31695 debug-util.cc:398] Leaking SignalData structure 0x7b08001dc820 after lost signal to thread 30649
W20260806 03:59:41.262053 31695 debug-util.cc:398] Leaking SignalData structure 0x7b08001dc900 after lost signal to thread 31551
W20260806 03:59:41.263098 31695 debug-util.cc:398] Leaking SignalData structure 0x7b08000e0920 after lost signal to thread 31701
W20260806 03:59:41.264032 31695 debug-util.cc:398] Leaking SignalData structure 0x7b08000dd020 after lost signal to thread 31815
W20260806 03:59:41.264962 31695 debug-util.cc:398] Leaking SignalData structure 0x7b080009e500 after lost signal to thread 31919
I20260806 03:59:41.284153 32003 consensus_queue.cc:1035] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.300484 32014 consensus_queue.cc:1035] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.313844 32014 consensus_queue.cc:1035] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.317857 31999 consensus_queue.cc:1035] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.357405 32014 consensus_queue.cc:1035] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.375744 32014 consensus_queue.cc:1035] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.442085 31994 consensus_queue.cc:1035] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:41.442818 32023 consensus_queue.cc:1035] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:59:42.052155 31574 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260806 03:59:42.169596 31914 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7280 after lost signal to thread 30649
W20260806 03:59:42.170342 31914 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7980 after lost signal to thread 31551
W20260806 03:59:42.171156 31914 debug-util.cc:398] Leaking SignalData structure 0x7b08001f18a0 after lost signal to thread 31701
W20260806 03:59:42.171886 31914 debug-util.cc:398] Leaking SignalData structure 0x7b08001813e0 after lost signal to thread 31815
W20260806 03:59:42.172605 31914 debug-util.cc:398] Leaking SignalData structure 0x7b0800196880 after lost signal to thread 31919
I20260806 03:59:42.308511 31999 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:59:42.309054 31999 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:42.311481 31999 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:42.311892 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" is_pre_election: true
I20260806 03:59:42.312605 31656 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 1.
I20260806 03:59:42.314190 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" is_pre_election: true
I20260806 03:59:42.314359 31832 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:42.314833 31763 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 1.
I20260806 03:59:42.316615 31999 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:59:42.317028 31999 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:59:42.317394 31999 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:42.324010 31999 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:42.326807 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:42.327013 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:42.327402 31656 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:42.327585 31763 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:42.334197 31763 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 2.
I20260806 03:59:42.334620 31656 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 2.
I20260806 03:59:42.335412 31833 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:42.336310 32073 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:42.336397 31999 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 2 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:42.337718 32073 raft_consensus.cc:686] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:42.338548 32073 consensus_queue.cc:237] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:42.348212 31493 catalog_manager.cc:5495] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 0 to 2, leader changed from <none> to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:42.742961 31656 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 03:59:42.744414 32073 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:42.769435 31763 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 03:59:42.770994 32073 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:59:42.917582 31874 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
mode: GRACEFUL
new_leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:42500
I20260806 03:59:42.918403 31874 raft_consensus.cc:595] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Received request to transfer leadership to TS c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:42.937182 31763 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
mode: GRACEFUL
new_leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:43106
I20260806 03:59:42.937860 31763 raft_consensus.cc:595] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 LEADER]: Received request to transfer leadership to TS c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:42.943532 31874 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d571ad6acd49444ab87adb4fec6d97ec"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
mode: GRACEFUL
new_leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:42500
I20260806 03:59:42.944221 31874 raft_consensus.cc:595] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Received request to transfer leadership to TS c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:42.969090 31653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
mode: GRACEFUL
new_leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:58678
I20260806 03:59:42.969772 31653 raft_consensus.cc:595] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Received request to transfer leadership to TS 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:42.975421 31653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
mode: GRACEFUL
new_leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:58678
I20260806 03:59:42.976017 31653 raft_consensus.cc:595] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Received request to transfer leadership to TS 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:42.982300 31874 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
mode: GRACEFUL
new_leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:42500
I20260806 03:59:42.982939 31874 raft_consensus.cc:595] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Received request to transfer leadership to TS 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:42.989962 31880 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
mode: GRACEFUL
new_leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:42500
I20260806 03:59:42.990568 31880 raft_consensus.cc:595] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Received request to transfer leadership to TS 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:43.070068 32098 raft_consensus.cc:979] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815: : Instructing follower c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:43.070596 32023 raft_consensus.cc:1067] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 LEADER]: Signalling peer c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:43.072540 31653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:58670
I20260806 03:59:43.073119 31653 raft_consensus.cc:482] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:43.073493 31653 raft_consensus.cc:3037] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.080313 31653 raft_consensus.cc:504] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.082252 31653 leader_election.cc:288] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 2 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:43.083246 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:43.083877 31763 raft_consensus.cc:3032] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:43.084210 31763 raft_consensus.cc:729] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:43.084987 31763 consensus_queue.cc:260] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.086112 31763 raft_consensus.cc:3037] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.089440 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdbef037e374999a11b9e55e263ea6f" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:43.090013 31880 raft_consensus.cc:3037] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.094689 31763 raft_consensus.cc:2454] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 2.
I20260806 03:59:43.096017 31606 leader_election.cc:302] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:43.096874 32014 raft_consensus.cc:2781] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:43.101212 31880 raft_consensus.cc:2454] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 2.
I20260806 03:59:43.101744 32014 raft_consensus.cc:686] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:43.102667 32014 consensus_queue.cc:237] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.111753 31493 catalog_manager.cc:5495] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 1 to 2, leader changed from 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130) to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:43.143071 32100 raft_consensus.cc:979] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79: : Instructing follower 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:43.143599 32014 raft_consensus.cc:1067] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Signalling peer 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:43.152634 32075 raft_consensus.cc:979] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018: : Instructing follower c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:43.153110 32075 raft_consensus.cc:1067] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Signalling peer c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:43.156028 31653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:58666
I20260806 03:59:43.156670 31653 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:43.157024 31653 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:43.161425 31763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:43098
I20260806 03:59:43.162047 31763 raft_consensus.cc:482] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:43.162410 31763 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.166232 31653 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.168982 31653 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:43.170421 31765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:43.171164 31765 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:43.172045 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:43.172631 31880 raft_consensus.cc:3032] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:43.172920 31880 raft_consensus.cc:729] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:43.173681 31880 consensus_queue.cc:260] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.174757 31880 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:43.177342 32100 raft_consensus.cc:979] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79: : Instructing follower 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:43.177750 32100 raft_consensus.cc:1067] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Signalling peer 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:43.180775 31765 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:43.182226 31767 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:43098
I20260806 03:59:43.182963 31767 raft_consensus.cc:482] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:43.183338 31767 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.183741 31999 raft_consensus.cc:979] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018: : Instructing follower c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:43.184196 32073 raft_consensus.cc:1067] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Signalling peer c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:43.185932 31656 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d571ad6acd49444ab87adb4fec6d97ec"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:58666
I20260806 03:59:43.186486 31656 raft_consensus.cc:482] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:43.186838 31656 raft_consensus.cc:3037] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.190233 31767 raft_consensus.cc:504] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.193192 31656 raft_consensus.cc:504] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.194716 31660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:43.195079 31656 leader_election.cc:288] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 2 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:43.195436 31660 raft_consensus.cc:3032] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:43.195768 31660 raft_consensus.cc:729] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:43.196282 31763 raft_consensus.cc:504] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.196532 31660 consensus_queue.cc:260] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.197700 31660 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.201049 31880 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:43.201522 31874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571ad6acd49444ab87adb4fec6d97ec" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:43.202101 31874 raft_consensus.cc:3032] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:43.202414 31874 raft_consensus.cc:729] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:43.203261 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:43.203171 31874 consensus_queue.cc:260] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.203866 31877 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.204830 31874 raft_consensus.cc:3037] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.210843 31877 raft_consensus.cc:2454] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 2.
I20260806 03:59:43.212023 31724 leader_election.cc:302] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:43.212939 32023 raft_consensus.cc:2781] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:43.191839 31606 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:43.213385 32023 raft_consensus.cc:686] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Becoming Leader. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:43.216363 32023 consensus_queue.cc:237] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.218667 32014 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:43.214570 31763 leader_election.cc:288] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 2 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:43.215180 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:43.215227 31660 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 2.
I20260806 03:59:43.219142 32014 raft_consensus.cc:686] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:43.220780 31656 raft_consensus.cc:3032] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:43.220639 32014 consensus_queue.cc:237] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.222828 31722 leader_election.cc:302] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:43.223712 32098 raft_consensus.cc:2781] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:43.221113 31656 raft_consensus.cc:729] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:43.226524 31874 raft_consensus.cc:2454] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 2.
I20260806 03:59:43.227420 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:43.228003 31880 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.229771 31765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571ad6acd49444ab87adb4fec6d97ec" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:43.236624 31765 raft_consensus.cc:3037] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.227654 31607 leader_election.cc:302] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:43.232175 31767 leader_election.cc:288] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 2 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:43.243752 32014 raft_consensus.cc:2781] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:43.243142 31494 catalog_manager.cc:5495] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 reported cstate change: term changed from 1 to 2, leader changed from c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129) to 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:43.244277 32014 raft_consensus.cc:686] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:43.245249 32014 consensus_queue.cc:237] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.248518 32098 raft_consensus.cc:686] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Becoming Leader. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:43.249344 32098 consensus_queue.cc:237] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.232949 31656 consensus_queue.cc:260] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.233311 31493 catalog_manager.cc:5495] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 2 to 3, leader changed from 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131) to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:43.253219 31656 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.235208 31880 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 2.
I20260806 03:59:43.243176 31765 raft_consensus.cc:2454] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 2.
I20260806 03:59:43.270356 31656 raft_consensus.cc:2454] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 2.
I20260806 03:59:43.276734 31494 catalog_manager.cc:5495] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 1 to 2, leader changed from 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131) to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:43.279743 31493 catalog_manager.cc:5495] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 reported cstate change: term changed from 1 to 2, leader changed from c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129) to 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:43.359598 32075 raft_consensus.cc:979] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018: : Instructing follower 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:43.360073 31999 raft_consensus.cc:1067] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Signalling peer 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:43.361720 31765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:43084
I20260806 03:59:43.362335 31765 raft_consensus.cc:482] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:43.362699 31765 raft_consensus.cc:3037] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.369252 31765 raft_consensus.cc:504] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.371997 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:43.372216 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c56d39f4124b1397e875d0ebfc74a4" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:43.372607 31656 raft_consensus.cc:3037] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.372826 31880 raft_consensus.cc:3032] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:43.373232 31880 raft_consensus.cc:729] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:43.373986 31880 consensus_queue.cc:260] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.375067 31880 raft_consensus.cc:3037] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.379133 31656 raft_consensus.cc:2454] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 2.
I20260806 03:59:43.380543 31722 leader_election.cc:302] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:43.381387 32098 raft_consensus.cc:2781] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:43.381808 32098 raft_consensus.cc:686] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Becoming Leader. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:43.382025 31880 raft_consensus.cc:2454] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 2.
I20260806 03:59:43.383235 31765 leader_election.cc:288] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 2 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:43.383510 32098 consensus_queue.cc:237] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.391765 31493 catalog_manager.cc:5495] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 reported cstate change: term changed from 1 to 2, leader changed from 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131) to 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:43.392880 32073 raft_consensus.cc:979] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018: : Instructing follower 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:43.393311 31999 raft_consensus.cc:1067] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Signalling peer 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:43.394892 31765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:43084
I20260806 03:59:43.395462 31765 raft_consensus.cc:482] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:43.395777 31765 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.403821 31765 raft_consensus.cc:504] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.406608 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:43.406806 31880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:43.407342 31656 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.407418 31880 raft_consensus.cc:3032] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:43.407825 31880 raft_consensus.cc:729] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:43.408607 31880 consensus_queue.cc:260] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.409708 31880 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:43.418128 31656 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 2.
I20260806 03:59:43.420109 31880 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 2.
I20260806 03:59:43.420159 31722 leader_election.cc:302] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:43.421582 32098 raft_consensus.cc:2781] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:43.422043 32098 raft_consensus.cc:686] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Becoming Leader. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:43.422847 32098 consensus_queue.cc:237] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:43.424337 31765 leader_election.cc:288] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 2 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:43.431491 31493 catalog_manager.cc:5495] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 reported cstate change: term changed from 1 to 2, leader changed from 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131) to 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:43.649443 31656 raft_consensus.cc:1261] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.650856 32098 consensus_queue.cc:1035] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.664263 31880 raft_consensus.cc:1261] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.673535 32023 consensus_queue.cc:1035] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.675616 31880 raft_consensus.cc:1261] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.678885 32023 consensus_queue.cc:1035] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.693930 31877 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:59:43.695619 32100 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.710675 31765 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:59:43.713027 32100 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.725363 31656 raft_consensus.cc:1261] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.733398 31874 raft_consensus.cc:1261] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.734920 32108 consensus_queue.cc:1035] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.736413 32098 consensus_queue.cc:1035] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.748392 31765 raft_consensus.cc:1261] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.749791 32014 consensus_queue.cc:1035] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:59:43.811244 31656 raft_consensus.cc:1261] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.812870 32113 consensus_queue.cc:1035] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.820962 31763 raft_consensus.cc:1261] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.824456 32108 consensus_queue.cc:1035] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.848235 31877 raft_consensus.cc:1261] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.849900 32108 consensus_queue.cc:1035] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:59:43.872066 31653 raft_consensus.cc:1261] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.873564 32023 consensus_queue.cc:1035] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.899677 31880 raft_consensus.cc:1261] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.901253 32098 consensus_queue.cc:1035] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:43.912501 31877 raft_consensus.cc:1261] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:43.914016 32098 consensus_queue.cc:1035] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:46.026930 30646 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260806 03:59:46.027649 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:46.054118 31653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
mode: GRACEFUL
new_leader_uuid: "2166aaf7ebfc46f5ab80e355af820018"
from {username='slave'} at 127.0.0.1:58682
I20260806 03:59:46.055022 31653 raft_consensus.cc:595] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Received request to transfer leadership to TS 2166aaf7ebfc46f5ab80e355af820018
I20260806 03:59:46.057809 31653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
mode: GRACEFUL
new_leader_uuid: "2166aaf7ebfc46f5ab80e355af820018"
from {username='slave'} at 127.0.0.1:58682
I20260806 03:59:46.058280 31653 raft_consensus.cc:595] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Received request to transfer leadership to TS 2166aaf7ebfc46f5ab80e355af820018
I20260806 03:59:46.059378 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:59:46.059790 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:46.174806 32146 raft_consensus.cc:979] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79: : Instructing follower 2166aaf7ebfc46f5ab80e355af820018 to start an election
I20260806 03:59:46.175218 32108 raft_consensus.cc:1067] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Signalling peer 2166aaf7ebfc46f5ab80e355af820018 to start an election
I20260806 03:59:46.176956 31877 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
from {username='slave'} at 127.0.0.1:42494
I20260806 03:59:46.177614 31877 raft_consensus.cc:482] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:46.178038 31877 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:46.184917 31877 raft_consensus.cc:504] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:46.187346 31877 leader_election.cc:288] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 2 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:46.188020 31653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:46.188560 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:46.188719 31653 raft_consensus.cc:3032] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:59:46.189138 31763 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:46.189167 31653 raft_consensus.cc:729] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:46.190318 31653 consensus_queue.cc:260] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:46.191679 31653 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:59:46.196324 31763 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 2.
I20260806 03:59:46.197690 31833 leader_election.cc:302] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:46.198549 31999 raft_consensus.cc:2781] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:59:46.199034 31653 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 2.
I20260806 03:59:46.199535 31999 raft_consensus.cc:686] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:46.200340 31999 consensus_queue.cc:237] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:46.209126 31493 catalog_manager.cc:5495] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 1 to 2, leader changed from c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129) to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:46.363662 32146 raft_consensus.cc:979] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79: : Instructing follower 2166aaf7ebfc46f5ab80e355af820018 to start an election
I20260806 03:59:46.364231 32108 raft_consensus.cc:1067] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Signalling peer 2166aaf7ebfc46f5ab80e355af820018 to start an election
I20260806 03:59:46.365998 31877 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
from {username='slave'} at 127.0.0.1:42494
I20260806 03:59:46.366648 31877 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:46.367063 31877 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:46.374125 31877 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:46.376297 31877 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 4 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:46.377244 31653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:46.377636 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:46.378099 31653 raft_consensus.cc:3032] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:59:46.378115 31763 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:46.378590 31653 raft_consensus.cc:729] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:46.379628 31653 consensus_queue.cc:260] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:46.380944 31653 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:46.385150 31763 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 4.
I20260806 03:59:46.386759 31833 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:46.387892 31999 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:59:46.388533 31999 raft_consensus.cc:686] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:46.389667 31999 consensus_queue.cc:237] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:46.390304 31653 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 4.
I20260806 03:59:46.399231 31493 catalog_manager.cc:5495] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 3 to 4, leader changed from c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129) to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 4 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:46.586552 31653 raft_consensus.cc:1261] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:46.587817 31999 consensus_queue.cc:1035] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:46.598749 31763 raft_consensus.cc:1261] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:59:46.600564 32148 consensus_queue.cc:1035] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:59:46.861414 31763 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:59:46.862999 32158 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:46.874981 31653 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:59:46.876996 32165 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:47.060680 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:47.066017 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:47.066416 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:48.067477 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:48.073469 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:48.073866 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:49.074694 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:49.079658 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:49.080050 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:50.080983 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:50.087260 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:50.087683 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:51.088645 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:51.096323 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:51.096815 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:52.097682 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:52.102989 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:52.103426 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:53.104213 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:53.109619 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:53.109993 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:54.110842 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:54.116642 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:54.117076 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:55.117865 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:55.124306 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:59:55.124727 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:56.153955 30646 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260806 03:59:56.158162 31877 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
mode: GRACEFUL
new_leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:42500
I20260806 03:59:56.158799 31877 raft_consensus.cc:595] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Received request to transfer leadership to TS c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:56.162983 31877 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
mode: GRACEFUL
new_leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:42500
I20260806 03:59:56.163362 31877 raft_consensus.cc:595] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Received request to transfer leadership to TS c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:56.173974 31767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
mode: GRACEFUL
new_leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:43106
I20260806 03:59:56.174448 31767 raft_consensus.cc:595] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Received request to transfer leadership to TS c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:56.178602 31767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
mode: GRACEFUL
new_leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:43106
I20260806 03:59:56.179090 31767 raft_consensus.cc:595] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Received request to transfer leadership to TS c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:56.183075 31767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
mode: GRACEFUL
new_leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:43106
I20260806 03:59:56.183542 31767 raft_consensus.cc:595] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Received request to transfer leadership to TS c7dd78b16d8842e88ec02c12a3a6ba79
I20260806 03:59:56.250641 32187 raft_consensus.cc:979] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815: : Instructing follower c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.251148 32181 raft_consensus.cc:1067] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Signalling peer c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.252532 31653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:58670
I20260806 03:59:56.252954 31653 raft_consensus.cc:482] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:56.253276 31653 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.258541 31653 raft_consensus.cc:504] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.260509 31653 leader_election.cc:288] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:56.261446 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:56.261471 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0370dede28aa4bdf98ad4b7fb852e561" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:56.262279 31767 raft_consensus.cc:3032] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:56.262323 31877 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.262599 31767 raft_consensus.cc:729] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:56.263311 31767 consensus_queue.cc:260] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.264364 31767 raft_consensus.cc:3037] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.267086 31877 raft_consensus.cc:2454] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:56.268129 31607 leader_election.cc:302] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:56.268829 32182 raft_consensus.cc:2781] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:56.269315 31767 raft_consensus.cc:2454] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:56.269778 32182 raft_consensus.cc:686] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:56.270664 32182 consensus_queue.cc:237] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.277675 31493 catalog_manager.cc:5495] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 2 to 3, leader changed from 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130) to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:56.410106 32183 raft_consensus.cc:979] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018: : Instructing follower c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.410638 32195 raft_consensus.cc:1067] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Signalling peer c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.412039 31653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:58666
I20260806 03:59:56.412456 31653 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:56.412676 31653 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:59:56.417323 31653 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.418677 31653 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 5 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:56.419371 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:56.419617 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:56.419860 31767 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:59:56.420179 31877 raft_consensus.cc:3032] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:59:56.420467 31877 raft_consensus.cc:729] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:56.421041 31877 consensus_queue.cc:260] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.421919 31877 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:59:56.425271 31767 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 5.
I20260806 03:59:56.426354 31606 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:56.426964 32182 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:59:56.427299 31877 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 5.
I20260806 03:59:56.427380 32182 raft_consensus.cc:686] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 5 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:56.428363 32182 consensus_queue.cc:237] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.430466 32187 raft_consensus.cc:979] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815: : Instructing follower c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.430886 32181 raft_consensus.cc:1067] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Signalling peer c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.432437 31653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:58670
I20260806 03:59:56.432937 31653 raft_consensus.cc:482] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:56.433290 31653 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.436363 31493 catalog_manager.cc:5495] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 4 to 5, leader changed from 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131) to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 5 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:56.439195 31653 raft_consensus.cc:504] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.441004 31653 leader_election.cc:288] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:56.441659 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:56.441920 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:56.442231 31767 raft_consensus.cc:3032] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:56.442404 31877 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.442569 31767 raft_consensus.cc:729] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:56.443368 31767 consensus_queue.cc:260] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.444470 31767 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.448534 31877 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:56.449404 31607 leader_election.cc:302] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:56.449975 32182 raft_consensus.cc:2781] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:56.450275 31767 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:56.450335 32182 raft_consensus.cc:686] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:56.451213 32182 consensus_queue.cc:237] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.457803 31493 catalog_manager.cc:5495] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 2 to 3, leader changed from 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130) to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:56.575124 32187 raft_consensus.cc:979] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815: : Instructing follower c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.575605 32181 raft_consensus.cc:1067] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Signalling peer c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.576921 31653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:58670
I20260806 03:59:56.577338 31653 raft_consensus.cc:482] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:56.577610 31653 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.581933 31653 raft_consensus.cc:504] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.583344 31653 leader_election.cc:288] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Requested vote from peers 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:56.584048 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:56.584313 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:56.584672 31767 raft_consensus.cc:3032] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:56.584751 31877 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.584966 31767 raft_consensus.cc:729] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:56.585723 31767 consensus_queue.cc:260] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.586699 31767 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.588829 31877 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:56.589609 31607 leader_election.cc:302] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:56.590200 32182 raft_consensus.cc:2781] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:56.590693 32182 raft_consensus.cc:686] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:56.591575 32182 consensus_queue.cc:237] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:56.592054 31767 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:56.598668 31493 catalog_manager.cc:5495] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 2 to 3, leader changed from 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130) to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:56.668601 32183 raft_consensus.cc:979] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018: : Instructing follower c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.669010 32195 raft_consensus.cc:1067] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Signalling peer c7dd78b16d8842e88ec02c12a3a6ba79 to start an election
I20260806 03:59:56.670246 31653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
from {username='slave'} at 127.0.0.1:58666
I20260806 03:59:56.670676 31653 raft_consensus.cc:482] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:56.670955 31653 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.675218 31653 raft_consensus.cc:504] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:56.676473 31653 leader_election.cc:288] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Requested vote from peers 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:56.677284 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:56.677389 31767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:56.677763 31877 raft_consensus.cc:3032] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:59:56.677915 31767 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.678048 31877 raft_consensus.cc:729] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:56.678689 31877 consensus_queue.cc:260] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:56.679507 31877 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:59:56.682471 31767 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:56.683339 31606 leader_election.cc:302] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6562a8a28a404b7b8dc7f9bdbe22f815, c7dd78b16d8842e88ec02c12a3a6ba79; no voters:
I20260806 03:59:56.683575 31877 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7dd78b16d8842e88ec02c12a3a6ba79 in term 3.
I20260806 03:59:56.683935 32182 raft_consensus.cc:2781] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:59:56.684358 32182 raft_consensus.cc:686] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Leader. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:56.685086 32182 consensus_queue.cc:237] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:56.692044 31493 catalog_manager.cc:5495] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 reported cstate change: term changed from 2 to 3, leader changed from 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131) to c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:56.706542 31877 raft_consensus.cc:1261] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:56.707571 32182 consensus_queue.cc:1035] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:56.720009 31767 raft_consensus.cc:1261] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:56.724010 32188 consensus_queue.cc:1035] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:59:56.869323 31767 raft_consensus.cc:1261] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:56.870618 32188 consensus_queue.cc:1035] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:56.879101 31877 raft_consensus.cc:1261] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:56.880252 32188 consensus_queue.cc:1035] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:57.003016 31877 raft_consensus.cc:1261] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:57.004225 32188 consensus_queue.cc:1035] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:57.013440 31767 raft_consensus.cc:1261] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:57.015355 32182 consensus_queue.cc:1035] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:57.019414 31763 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 5 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:59:57.020934 32182 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:59:57.036237 31877 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 5 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:59:57.039255 32215 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:59:57.108392 31763 raft_consensus.cc:1261] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:57.109510 32215 consensus_queue.cc:1035] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:59:57.119733 31877 raft_consensus.cc:1261] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Refusing update from remote peer c7dd78b16d8842e88ec02c12a3a6ba79: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:59:57.121081 32188 consensus_queue.cc:1035] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:59:59.216873 30646 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260806 03:59:59.217705 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:59:59.224990 31653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
mode: GRACEFUL
new_leader_uuid: "2166aaf7ebfc46f5ab80e355af820018"
from {username='slave'} at 127.0.0.1:58682
I20260806 03:59:59.225622 31653 raft_consensus.cc:595] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Received request to transfer leadership to TS 2166aaf7ebfc46f5ab80e355af820018
I20260806 03:59:59.227918 31653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
mode: GRACEFUL
new_leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:58682
I20260806 03:59:59.228379 31653 raft_consensus.cc:595] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Received request to transfer leadership to TS 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:59.230312 31653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
mode: GRACEFUL
new_leader_uuid: "2166aaf7ebfc46f5ab80e355af820018"
from {username='slave'} at 127.0.0.1:58682
I20260806 03:59:59.230716 31653 raft_consensus.cc:595] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Received request to transfer leadership to TS 2166aaf7ebfc46f5ab80e355af820018
I20260806 03:59:59.232587 31653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4"
dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
mode: GRACEFUL
new_leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:58682
I20260806 03:59:59.232949 31653 raft_consensus.cc:595] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 5 LEADER]: Received request to transfer leadership to TS 6562a8a28a404b7b8dc7f9bdbe22f815
I20260806 03:59:59.233713 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 03:59:59.234151 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:59:59.378582 32238 raft_consensus.cc:979] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79: : Instructing follower 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:59.379204 32239 raft_consensus.cc:1067] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Signalling peer 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:59.380793 31763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:43098
I20260806 03:59:59.381237 31763 raft_consensus.cc:482] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:59.381539 31763 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.386883 31763 raft_consensus.cc:504] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.388855 31763 leader_election.cc:288] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 4 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:59.389834 31653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:59.389972 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b777807a4fe4cb9941725b6390218f3" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:59.390520 31653 raft_consensus.cc:3032] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:59:59.390579 31877 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.390799 31653 raft_consensus.cc:729] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:59.391485 31653 consensus_queue.cc:260] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.392503 31653 raft_consensus.cc:3037] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.395401 31877 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 4.
I20260806 03:59:59.396445 31724 leader_election.cc:302] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:59.397204 32181 raft_consensus.cc:2781] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:59:59.397382 31653 raft_consensus.cc:2454] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 4.
I20260806 03:59:59.398114 32181 raft_consensus.cc:686] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 LEADER]: Becoming Leader. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:59.398933 32181 consensus_queue.cc:237] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.405558 31488 catalog_manager.cc:5495] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 reported cstate change: term changed from 3 to 4, leader changed from c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129) to 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130). New cstate: current_term: 4 leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:59.436812 32238 raft_consensus.cc:979] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79: : Instructing follower 2166aaf7ebfc46f5ab80e355af820018 to start an election
I20260806 03:59:59.437244 32239 raft_consensus.cc:1067] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Signalling peer 2166aaf7ebfc46f5ab80e355af820018 to start an election
I20260806 03:59:59.438560 31877 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
from {username='slave'} at 127.0.0.1:42494
I20260806 03:59:59.439023 31877 raft_consensus.cc:482] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:59.439298 31877 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.443780 31877 raft_consensus.cc:504] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.445648 31877 leader_election.cc:288] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 4 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:59.446400 31653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:59.446633 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ef48c15e7f49f0a55a11ffed3bcf0d" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:59.446905 31653 raft_consensus.cc:3032] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:59:59.447127 31763 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.447176 31653 raft_consensus.cc:729] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:59.447902 31653 consensus_queue.cc:260] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.448819 31653 raft_consensus.cc:3037] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.451529 31763 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 4.
I20260806 03:59:59.452560 31833 leader_election.cc:302] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:59.453128 31653 raft_consensus.cc:2454] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 4.
I20260806 03:59:59.453281 32236 raft_consensus.cc:2781] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:59:59.454546 32236 raft_consensus.cc:686] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:59.455475 32236 consensus_queue.cc:237] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.462514 31488 catalog_manager.cc:5495] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 3 to 4, leader changed from c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129) to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 4 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } }
I20260806 03:59:59.515339 32238 raft_consensus.cc:979] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79: : Instructing follower 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:59.515760 32239 raft_consensus.cc:1067] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 5 LEADER]: Signalling peer 6562a8a28a404b7b8dc7f9bdbe22f815 to start an election
I20260806 03:59:59.517235 31763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4"
dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
from {username='slave'} at 127.0.0.1:43098
I20260806 03:59:59.517695 31763 raft_consensus.cc:482] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:59.517984 31763 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:59:59.522143 31763 raft_consensus.cc:504] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.523423 31763 leader_election.cc:288] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 6 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131:35351)
I20260806 03:59:59.524363 31653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:59.524501 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b0116d16a04a3f8012c44fc8c94be4" candidate_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
I20260806 03:59:59.524812 31653 raft_consensus.cc:3032] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:59:59.524968 31877 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:59:59.525077 31653 raft_consensus.cc:729] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:59.525728 31653 consensus_queue.cc:260] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.526628 31653 raft_consensus.cc:3037] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:59:59.530205 31877 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 6.
I20260806 03:59:59.531255 31724 leader_election.cc:302] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:59.531952 32181 raft_consensus.cc:2781] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:59:59.532433 32181 raft_consensus.cc:686] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 6 LEADER]: Becoming Leader. State: Replica: 6562a8a28a404b7b8dc7f9bdbe22f815, State: Running, Role: LEADER
I20260806 03:59:59.532481 31653 raft_consensus.cc:2454] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6562a8a28a404b7b8dc7f9bdbe22f815 in term 6.
I20260806 03:59:59.533396 32181 consensus_queue.cc:237] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } }
I20260806 03:59:59.540359 31488 catalog_manager.cc:5495] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 reported cstate change: term changed from 5 to 6, leader changed from c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129) to 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130). New cstate: current_term: 6 leader_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:59.748801 32239 raft_consensus.cc:979] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79: : Instructing follower 2166aaf7ebfc46f5ab80e355af820018 to start an election
I20260806 03:59:59.749233 32238 raft_consensus.cc:1067] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Signalling peer 2166aaf7ebfc46f5ab80e355af820018 to start an election
I20260806 03:59:59.750501 31877 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc"
dest_uuid: "2166aaf7ebfc46f5ab80e355af820018"
from {username='slave'} at 127.0.0.1:42494
I20260806 03:59:59.750905 31877 raft_consensus.cc:482] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:59:59.751165 31877 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.755410 31877 raft_consensus.cc:504] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:59.756713 31877 leader_election.cc:288] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 4 election: Requested vote from peers c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197), 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307)
I20260806 03:59:59.757496 31653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79"
I20260806 03:59:59.757601 31763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13da11470a074eb6be303c3ab41bd5cc" candidate_uuid: "2166aaf7ebfc46f5ab80e355af820018" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815"
I20260806 03:59:59.757920 31653 raft_consensus.cc:3032] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:59:59.758106 31763 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.758203 31653 raft_consensus.cc:729] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c7dd78b16d8842e88ec02c12a3a6ba79, State: Running, Role: LEADER
I20260806 03:59:59.758817 31653 consensus_queue.cc:260] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:59.759665 31653 raft_consensus.cc:3037] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:59:59.762166 31763 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 4.
I20260806 03:59:59.762964 31833 leader_election.cc:302] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2166aaf7ebfc46f5ab80e355af820018, 6562a8a28a404b7b8dc7f9bdbe22f815; no voters:
I20260806 03:59:59.763615 32236 raft_consensus.cc:2781] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:59:59.763589 31653 raft_consensus.cc:2454] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2166aaf7ebfc46f5ab80e355af820018 in term 4.
I20260806 03:59:59.764192 32236 raft_consensus.cc:686] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Becoming Leader. State: Replica: 2166aaf7ebfc46f5ab80e355af820018, State: Running, Role: LEADER
I20260806 03:59:59.765026 32236 consensus_queue.cc:237] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } }
I20260806 03:59:59.771473 31488 catalog_manager.cc:5495] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 reported cstate change: term changed from 3 to 4, leader changed from c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129) to 2166aaf7ebfc46f5ab80e355af820018 (127.29.237.131). New cstate: current_term: 4 leader_uuid: "2166aaf7ebfc46f5ab80e355af820018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260806 03:59:59.849339 31877 raft_consensus.cc:1261] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:59.850613 32181 consensus_queue.cc:1035] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:59.864014 31653 raft_consensus.cc:1261] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:59.865335 32259 consensus_queue.cc:1035] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:59.945209 31763 raft_consensus.cc:1261] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:59.946352 32236 consensus_queue.cc:1035] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:59:59.955677 31653 raft_consensus.cc:1261] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:59:59.957166 32250 consensus_queue.cc:1035] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:00.070590 31653 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 6 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:00:00.071779 32259 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:00.079851 31877 raft_consensus.cc:1261] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 6 FOLLOWER]: Refusing update from remote peer 6562a8a28a404b7b8dc7f9bdbe22f815: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:00:00.081691 32240 consensus_queue.cc:1035] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2166aaf7ebfc46f5ab80e355af820018" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:00.234975 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:00.240445 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:00.240854 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:00.389683 31763 raft_consensus.cc:1261] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:00:00.391088 32265 consensus_queue.cc:1035] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6562a8a28a404b7b8dc7f9bdbe22f815" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:00.392459 31653 raft_consensus.cc:1261] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Refusing update from remote peer 2166aaf7ebfc46f5ab80e355af820018: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:00:00.393595 32265 consensus_queue.cc:1035] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7dd78b16d8842e88ec02c12a3a6ba79" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:01.241703 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:01.248020 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:01.248435 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:02.249168 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:02.254418 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:02.254804 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:03.255626 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:03.260897 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:03.261324 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:04.262069 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:04.267201 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:04.267601 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:05.268457 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:05.273614 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:05.274047 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:06.274860 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:06.280501 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:06.280890 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:07.281630 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:07.287802 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:07.288239 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:08.289580 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:08.297757 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 04:00:08.298168 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:09.326890 30646 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260806 04:00:09.327417 30646 tablet_server.cc:178] TabletServer@127.29.237.129:0 shutting down...
I20260806 04:00:09.356887 30646 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:00:09.357595 30646 tablet_replica.cc:330] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.358274 30646 raft_consensus.cc:2229] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.358848 30646 raft_consensus.cc:2258] T 99ef48c15e7f49f0a55a11ffed3bcf0d P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.360899 30646 tablet_replica.cc:330] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.361392 30646 raft_consensus.cc:2229] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.362228 30646 raft_consensus.cc:2258] T 0cdbef037e374999a11b9e55e263ea6f P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.364153 30646 tablet_replica.cc:330] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.364652 30646 raft_consensus.cc:2229] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.365064 30646 raft_consensus.cc:2258] T 13da11470a074eb6be303c3ab41bd5cc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.366935 30646 tablet_replica.cc:330] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.367419 30646 raft_consensus.cc:2229] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.367815 30646 raft_consensus.cc:2258] T 3b777807a4fe4cb9941725b6390218f3 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.369591 30646 tablet_replica.cc:330] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.370182 30646 raft_consensus.cc:2229] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.371130 30646 raft_consensus.cc:2258] T 0370dede28aa4bdf98ad4b7fb852e561 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.372900 30646 tablet_replica.cc:330] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.373358 30646 raft_consensus.cc:2229] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.374118 30646 raft_consensus.cc:2258] T d571ad6acd49444ab87adb4fec6d97ec P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.375895 30646 tablet_replica.cc:330] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.376369 30646 raft_consensus.cc:2229] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.376792 30646 raft_consensus.cc:2258] T e0c56d39f4124b1397e875d0ebfc74a4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.378468 30646 tablet_replica.cc:330] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.379006 30646 raft_consensus.cc:2229] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.379396 30646 raft_consensus.cc:2258] T bdc88796a69d4798bffcf0cbeb934ebc P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.381036 30646 tablet_replica.cc:330] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.381466 30646 raft_consensus.cc:2229] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.382227 30646 raft_consensus.cc:2258] T 54c76bab5b5d4876a48b6914768628aa P c7dd78b16d8842e88ec02c12a3a6ba79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.383878 30646 tablet_replica.cc:330] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79: stopping tablet replica
I20260806 04:00:09.384352 30646 raft_consensus.cc:2229] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.384739 30646 raft_consensus.cc:2258] T c4b0116d16a04a3f8012c44fc8c94be4 P c7dd78b16d8842e88ec02c12a3a6ba79 [term 6 FOLLOWER]: Raft consensus is shut down!
W20260806 04:00:09.400624 31607 connection.cc:441] server connection from 127.0.0.1:58666 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58666 (request call id 191) could send its response
W20260806 04:00:09.401641 31832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.237.129:37197 (error 108) [suppressed 6 similar messages]
W20260806 04:00:09.405853 31832 consensus_peers.cc:487] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 -> Peer c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197): Couldn't send request to peer c7dd78b16d8842e88ec02c12a3a6ba79. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.129:37197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:09.434681 30646 tablet_server.cc:194] TabletServer@127.29.237.129:0 shutdown complete.
W20260806 04:00:09.438252 31722 consensus_peers.cc:487] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 -> Peer c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197): Couldn't send request to peer c7dd78b16d8842e88ec02c12a3a6ba79. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.129:37197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:09.454058 30646 tablet_server.cc:178] TabletServer@127.29.237.130:0 shutting down...
W20260806 04:00:09.473757 31832 consensus_peers.cc:487] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 -> Peer c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197): Couldn't send request to peer c7dd78b16d8842e88ec02c12a3a6ba79. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.129:37197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:09.482267 30646 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:00:09.482897 30646 tablet_replica.cc:330] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.483546 30646 raft_consensus.cc:2229] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.484115 30646 raft_consensus.cc:2258] T 13da11470a074eb6be303c3ab41bd5cc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.485908 30646 tablet_replica.cc:330] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.486428 30646 raft_consensus.cc:2229] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 6 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.487236 30646 raft_consensus.cc:2258] T c4b0116d16a04a3f8012c44fc8c94be4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.488960 30646 tablet_replica.cc:330] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.489455 30646 raft_consensus.cc:2229] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.489847 30646 raft_consensus.cc:2258] T 0370dede28aa4bdf98ad4b7fb852e561 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.491504 30646 tablet_replica.cc:330] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.492025 30646 raft_consensus.cc:2229] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.492854 30646 raft_consensus.cc:2258] T 3b777807a4fe4cb9941725b6390218f3 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.494449 30646 tablet_replica.cc:330] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.494904 30646 raft_consensus.cc:2229] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.495308 30646 raft_consensus.cc:2258] T 0cdbef037e374999a11b9e55e263ea6f P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.496970 30646 tablet_replica.cc:330] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.497421 30646 raft_consensus.cc:2229] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.497817 30646 raft_consensus.cc:2258] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.499398 30646 tablet_replica.cc:330] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.499836 30646 raft_consensus.cc:2229] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.500234 30646 raft_consensus.cc:2258] T bdc88796a69d4798bffcf0cbeb934ebc P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.501767 30646 tablet_replica.cc:330] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.502189 30646 raft_consensus.cc:2229] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.502573 30646 raft_consensus.cc:2258] T 54c76bab5b5d4876a48b6914768628aa P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.504639 30646 tablet_replica.cc:330] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.505097 30646 raft_consensus.cc:2229] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 LEADER]: Raft consensus shutting down.
W20260806 04:00:09.505617 32259 consensus_queue.cc:1159] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2166aaf7ebfc46f5ab80e355af820018" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20260806 04:00:09.506754 30646 raft_consensus.cc:2258] T e0c56d39f4124b1397e875d0ebfc74a4 P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.508446 30646 tablet_replica.cc:330] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815: stopping tablet replica
I20260806 04:00:09.508857 30646 raft_consensus.cc:2229] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.509173 30646 raft_consensus.cc:2258] T d571ad6acd49444ab87adb4fec6d97ec P 6562a8a28a404b7b8dc7f9bdbe22f815 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.537696 30646 tablet_server.cc:194] TabletServer@127.29.237.130:0 shutdown complete.
I20260806 04:00:09.557490 30646 tablet_server.cc:178] TabletServer@127.29.237.131:0 shutting down...
W20260806 04:00:09.573047 31832 consensus_peers.cc:487] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 -> Peer c7dd78b16d8842e88ec02c12a3a6ba79 (127.29.237.129:37197): Couldn't send request to peer c7dd78b16d8842e88ec02c12a3a6ba79. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.129:37197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:09.586002 30646 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 04:00:09.586843 30646 tablet_replica.cc:330] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.587572 30646 raft_consensus.cc:2229] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.588021 30646 raft_consensus.cc:2258] T 0cdbef037e374999a11b9e55e263ea6f P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.590030 30646 tablet_replica.cc:330] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
W20260806 04:00:09.590106 31833 consensus_peers.cc:487] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 -> Peer 6562a8a28a404b7b8dc7f9bdbe22f815 (127.29.237.130:42307): Couldn't send request to peer 6562a8a28a404b7b8dc7f9bdbe22f815. Status: Network error: Client connection negotiation failed: client connection to 127.29.237.130:42307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:09.590626 30646 raft_consensus.cc:2229] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.591600 30646 raft_consensus.cc:2258] T 99ef48c15e7f49f0a55a11ffed3bcf0d P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.593626 30646 tablet_replica.cc:330] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.594085 30646 raft_consensus.cc:2229] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.594437 30646 raft_consensus.cc:2258] T 3b777807a4fe4cb9941725b6390218f3 P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.596009 30646 tablet_replica.cc:330] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.596517 30646 raft_consensus.cc:2229] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.596872 30646 raft_consensus.cc:2258] T 0370dede28aa4bdf98ad4b7fb852e561 P 2166aaf7ebfc46f5ab80e355af820018 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.598376 30646 tablet_replica.cc:330] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.598788 30646 raft_consensus.cc:2229] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.599125 30646 raft_consensus.cc:2258] T 54c76bab5b5d4876a48b6914768628aa P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.600638 30646 tablet_replica.cc:330] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.601087 30646 raft_consensus.cc:2229] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.601424 30646 raft_consensus.cc:2258] T e0c56d39f4124b1397e875d0ebfc74a4 P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.602887 30646 tablet_replica.cc:330] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.603297 30646 raft_consensus.cc:2229] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.604024 30646 raft_consensus.cc:2258] T bdc88796a69d4798bffcf0cbeb934ebc P 2166aaf7ebfc46f5ab80e355af820018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.605643 30646 tablet_replica.cc:330] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.606092 30646 raft_consensus.cc:2229] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.606515 30646 raft_consensus.cc:2258] T c4b0116d16a04a3f8012c44fc8c94be4 P 2166aaf7ebfc46f5ab80e355af820018 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.608059 30646 tablet_replica.cc:330] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.608513 30646 raft_consensus.cc:2229] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:09.608872 30646 raft_consensus.cc:2258] T d571ad6acd49444ab87adb4fec6d97ec P 2166aaf7ebfc46f5ab80e355af820018 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.610427 30646 tablet_replica.cc:330] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018: stopping tablet replica
I20260806 04:00:09.610839 30646 raft_consensus.cc:2229] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 4 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.611568 30646 raft_consensus.cc:2258] T 13da11470a074eb6be303c3ab41bd5cc P 2166aaf7ebfc46f5ab80e355af820018 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.656584 30646 tablet_server.cc:194] TabletServer@127.29.237.131:0 shutdown complete.
I20260806 04:00:09.674957 30646 master.cc:537] Master@127.29.237.190:38723 shutting down...
I20260806 04:00:09.698724 30646 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:00:09.699287 30646 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:09.699591 30646 tablet_replica.cc:330] T 00000000000000000000000000000000 P 78614853f6fd40a7a8a2360261e5cca3: stopping tablet replica
I20260806 04:00:09.718596 30646 master.cc:552] Master@127.29.237.190:38723 shutdown complete.
2166aaf7ebfc46f5ab80e355af820018 1
6562a8a28a404b7b8dc7f9bdbe22f815 4
c7dd78b16d8842e88ec02c12a3a6ba79 5
2166aaf7ebfc46f5ab80e355af820018 3
6562a8a28a404b7b8dc7f9bdbe22f815 4
c7dd78b16d8842e88ec02c12a3a6ba79 3
2166aaf7ebfc46f5ab80e355af820018 1
6562a8a28a404b7b8dc7f9bdbe22f815 1
c7dd78b16d8842e88ec02c12a3a6ba79 8
2166aaf7ebfc46f5ab80e355af820018 3
6562a8a28a404b7b8dc7f9bdbe22f815 3
c7dd78b16d8842e88ec02c12a3a6ba79 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30845 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260806 04:00:09.757157 30646 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.237.190:40089
I20260806 04:00:09.758379 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:00:09.764448 32289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:09.764448 32287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:09.764801 32291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:00:09.766083 30646 server_base.cc:1034] running on GCE node
I20260806 04:00:09.767151 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:00:09.767334 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:00:09.767474 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988809767460 us; error 0 us; skew 500 ppm
I20260806 04:00:09.767951 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:00:09.773734 30646 webserver.cc:460] Webserver started at http://127.29.237.190:38097/ using document root <none> and password file <none>
I20260806 04:00:09.774195 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 04:00:09.774363 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:00:09.774629 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:00:09.775802 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/master-0-root/instance:
uuid: "a4f6b68aa31d452e8d84505ace50b57d"
format_stamp: "Formatted at 2026-08-06 04:00:09 on dist-test-slave-lgr5"
I20260806 04:00:09.780114 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:00:09.783250 32301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:09.783921 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:00:09.784214 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/master-0-root
uuid: "a4f6b68aa31d452e8d84505ace50b57d"
format_stamp: "Formatted at 2026-08-06 04:00:09 on dist-test-slave-lgr5"
I20260806 04:00:09.784523 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:00:09.807560 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:00:09.808844 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:00:09.846849 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.190:40089
I20260806 04:00:09.846933 32373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.190:40089 every 8 connection(s)
I20260806 04:00:09.850653 32374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:09.860697 32374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Bootstrap starting.
I20260806 04:00:09.865013 32374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:09.869252 32374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: No bootstrap required, opened a new log
I20260806 04:00:09.871282 32374 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f6b68aa31d452e8d84505ace50b57d" member_type: VOTER }
I20260806 04:00:09.871750 32374 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:09.871994 32374 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f6b68aa31d452e8d84505ace50b57d, State: Initialized, Role: FOLLOWER
I20260806 04:00:09.872651 32374 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f6b68aa31d452e8d84505ace50b57d" member_type: VOTER }
I20260806 04:00:09.873098 32374 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:00:09.873330 32374 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:00:09.873598 32374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:09.877892 32374 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f6b68aa31d452e8d84505ace50b57d" member_type: VOTER }
I20260806 04:00:09.878427 32374 leader_election.cc:302] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4f6b68aa31d452e8d84505ace50b57d; no voters:
I20260806 04:00:09.879657 32374 leader_election.cc:288] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:00:09.879976 32380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:09.881373 32380 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [term 1 LEADER]: Becoming Leader. State: Replica: a4f6b68aa31d452e8d84505ace50b57d, State: Running, Role: LEADER
I20260806 04:00:09.881984 32380 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f6b68aa31d452e8d84505ace50b57d" member_type: VOTER }
I20260806 04:00:09.882550 32374 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:00:09.886266 32382 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4f6b68aa31d452e8d84505ace50b57d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f6b68aa31d452e8d84505ace50b57d" member_type: VOTER } }
I20260806 04:00:09.886617 32383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4f6b68aa31d452e8d84505ace50b57d. Latest consensus state: current_term: 1 leader_uuid: "a4f6b68aa31d452e8d84505ace50b57d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f6b68aa31d452e8d84505ace50b57d" member_type: VOTER } }
I20260806 04:00:09.886991 32382 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [sys.catalog]: This master's current role is: LEADER
I20260806 04:00:09.887261 32383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d [sys.catalog]: This master's current role is: LEADER
I20260806 04:00:09.889583 32394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:00:09.895849 32394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:00:09.896950 30646 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 04:00:09.904497 32394 catalog_manager.cc:1347] Generated new cluster ID: 1ed32e83694843e2b24937b56549cb4f
I20260806 04:00:09.904783 32394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:00:09.918329 32394 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:00:09.919929 32394 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:00:09.941545 32394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Generated new TSK 0
I20260806 04:00:09.942243 32394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:00:09.964072 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:00:09.970283 32414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:09.972206 32416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:09.973199 32420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:00:09.973403 30646 server_base.cc:1034] running on GCE node
I20260806 04:00:09.974786 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:00:09.975021 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:00:09.975230 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988809975213 us; error 0 us; skew 500 ppm
I20260806 04:00:09.975903 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:00:09.978878 30646 webserver.cc:460] Webserver started at http://127.29.237.129:34165/ using document root <none> and password file <none>
I20260806 04:00:09.979310 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 04:00:09.979460 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:00:09.979662 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:00:09.980862 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/instance:
uuid: "ae0b3e43e7a64f789e51018fd5518d23"
format_stamp: "Formatted at 2026-08-06 04:00:09 on dist-test-slave-lgr5"
I20260806 04:00:09.985626 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:00:09.988876 32426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:09.989578 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:00:09.989838 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root
uuid: "ae0b3e43e7a64f789e51018fd5518d23"
format_stamp: "Formatted at 2026-08-06 04:00:09 on dist-test-slave-lgr5"
I20260806 04:00:09.990106 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:00:10.019551 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:00:10.020915 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:00:10.022353 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:00:10.024663 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:00:10.024852 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:10.025101 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:00:10.025260 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:10.064941 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.129:35941
I20260806 04:00:10.065045 32530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.129:35941 every 8 connection(s)
I20260806 04:00:10.069254 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:00:10.076712 32538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:10.078809 32539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:10.079929 32541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:00:10.081950 30646 server_base.cc:1034] running on GCE node
I20260806 04:00:10.083864 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:00:10.084144 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:00:10.084319 32532 heartbeater.cc:347] Connected to a master server at 127.29.237.190:40089
I20260806 04:00:10.084340 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988810084327 us; error 0 us; skew 500 ppm
I20260806 04:00:10.084831 32532 heartbeater.cc:464] Registering TS with master...
I20260806 04:00:10.085210 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:00:10.085624 32532 heartbeater.cc:511] Master 127.29.237.190:40089 requested a full tablet report, sending...
I20260806 04:00:10.087862 32326 ts_manager.cc:194] Registered new tserver with Master: ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:10.088239 30646 webserver.cc:460] Webserver started at http://127.29.237.130:44413/ using document root <none> and password file <none>
I20260806 04:00:10.088733 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 04:00:10.088902 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:00:10.089140 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:00:10.089828 32326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39754
I20260806 04:00:10.090373 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/instance:
uuid: "dd7344d85a154539b7f8c484f2a76772"
format_stamp: "Formatted at 2026-08-06 04:00:10 on dist-test-slave-lgr5"
I20260806 04:00:10.095158 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:00:10.098347 32550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:10.099087 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:00:10.099342 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root
uuid: "dd7344d85a154539b7f8c484f2a76772"
format_stamp: "Formatted at 2026-08-06 04:00:10 on dist-test-slave-lgr5"
I20260806 04:00:10.099606 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:00:10.111353 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:00:10.112557 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:00:10.113909 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:00:10.116031 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:00:10.116235 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:10.116451 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:00:10.116597 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:10.156352 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.130:37857
I20260806 04:00:10.156502 32659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.130:37857 every 8 connection(s)
I20260806 04:00:10.160867 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:00:10.167832 32665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:10.169948 32667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:00:10.171460 30646 server_base.cc:1034] running on GCE node
W20260806 04:00:10.172353 32669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:00:10.173229 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:00:10.173434 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:00:10.173606 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988810173593 us; error 0 us; skew 500 ppm
I20260806 04:00:10.174219 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:00:10.175048 32660 heartbeater.cc:347] Connected to a master server at 127.29.237.190:40089
I20260806 04:00:10.175450 32660 heartbeater.cc:464] Registering TS with master...
I20260806 04:00:10.176200 32660 heartbeater.cc:511] Master 127.29.237.190:40089 requested a full tablet report, sending...
I20260806 04:00:10.176885 30646 webserver.cc:460] Webserver started at http://127.29.237.131:43731/ using document root <none> and password file <none>
I20260806 04:00:10.177465 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 04:00:10.177659 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:00:10.177973 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:00:10.178256 32326 ts_manager.cc:194] Registered new tserver with Master: dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:10.179554 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/instance:
uuid: "f7c202b562ce482db283463d7f02e136"
format_stamp: "Formatted at 2026-08-06 04:00:10 on dist-test-slave-lgr5"
I20260806 04:00:10.180341 32326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39766
I20260806 04:00:10.184655 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:00:10.187915 32676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:10.188791 30646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:00:10.189137 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root
uuid: "f7c202b562ce482db283463d7f02e136"
format_stamp: "Formatted at 2026-08-06 04:00:10 on dist-test-slave-lgr5"
I20260806 04:00:10.189404 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:00:10.217772 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:00:10.218986 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:00:10.220399 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:00:10.222607 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:00:10.222783 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:10.223023 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:00:10.223161 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:10.262019 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.131:34975
I20260806 04:00:10.262132 326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.131:34975 every 8 connection(s)
I20260806 04:00:10.276727 327 heartbeater.cc:347] Connected to a master server at 127.29.237.190:40089
I20260806 04:00:10.277154 327 heartbeater.cc:464] Registering TS with master...
I20260806 04:00:10.277843 327 heartbeater.cc:511] Master 127.29.237.190:40089 requested a full tablet report, sending...
I20260806 04:00:10.279621 32326 ts_manager.cc:194] Registered new tserver with Master: f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:10.280591 30646 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015606612s
I20260806 04:00:10.281448 32326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39782
I20260806 04:00:10.283435 30646 test_util.cc:268] Using random seed: -325314156
I20260806 04:00:10.308332 32326 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39784:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:00:10.314260 32326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:00:10.508569 32600 tablet_service.cc:1467] Processing CreateTablet for tablet fb31133082b14fb889780e53b990e965 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 04:00:10.510064 32600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb31133082b14fb889780e53b990e965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.512925 32595 tablet_service.cc:1467] Processing CreateTablet for tablet db65aabb33f8427593eac56e96ca9cc6 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 04:00:10.515923 32592 tablet_service.cc:1467] Processing CreateTablet for tablet db915af860484020942e9495eda28804 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 04:00:10.517197 32598 tablet_service.cc:1467] Processing CreateTablet for tablet 42a553c7b71d493284e5e79b6f19f848 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 04:00:10.518136 32591 tablet_service.cc:1467] Processing CreateTablet for tablet 8417acc05db341aca15704cd05736651 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 04:00:10.519995 32594 tablet_service.cc:1467] Processing CreateTablet for tablet c271fbb1b29c41b8afbfd003a9c9cb3b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 04:00:10.510952 32596 tablet_service.cc:1467] Processing CreateTablet for tablet bf5ed335f5bb4cf491d1d169c809b5d9 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 04:00:10.522244 32595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db65aabb33f8427593eac56e96ca9cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.522907 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8417acc05db341aca15704cd05736651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.523507 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf5ed335f5bb4cf491d1d169c809b5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.524223 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a553c7b71d493284e5e79b6f19f848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.524259 32465 tablet_service.cc:1467] Processing CreateTablet for tablet fb31133082b14fb889780e53b990e965 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 04:00:10.525645 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb31133082b14fb889780e53b990e965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.525425 32589 tablet_service.cc:1467] Processing CreateTablet for tablet bd55a7d68ee84cb7a69c962f984f9372 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 04:00:10.526750 32589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd55a7d68ee84cb7a69c962f984f9372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.528414 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db915af860484020942e9495eda28804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.530918 32594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c271fbb1b29c41b8afbfd003a9c9cb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.538748 32587 tablet_service.cc:1467] Processing CreateTablet for tablet 65c8a38c9f554ef887e7782aa21ad891 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 04:00:10.540218 32587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c8a38c9f554ef887e7782aa21ad891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.555455 32585 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7858566354435c9b6b09c5b8c1714c (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 04:00:10.556888 32585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7858566354435c9b6b09c5b8c1714c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.562153 32598 tablet_service.cc:1467] Processing CreateTablet for tablet bec6dcc3e9784bbe83c0ee7452223cb4 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 04:00:10.564186 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec6dcc3e9784bbe83c0ee7452223cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.573808 32462 tablet_service.cc:1467] Processing CreateTablet for tablet 42a553c7b71d493284e5e79b6f19f848 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 04:00:10.575218 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a553c7b71d493284e5e79b6f19f848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.585386 32598 tablet_service.cc:1467] Processing CreateTablet for tablet d2ad471021a947e9a45d57c7be704453 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 04:00:10.586884 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ad471021a947e9a45d57c7be704453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.590006 32461 tablet_service.cc:1467] Processing CreateTablet for tablet bf5ed335f5bb4cf491d1d169c809b5d9 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 04:00:10.591341 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf5ed335f5bb4cf491d1d169c809b5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.598097 351 tablet_bootstrap.cc:492] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:10.602774 32591 tablet_service.cc:1467] Processing CreateTablet for tablet 46d78e8748804374b8af237c1ca8ede8 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 04:00:10.603112 32598 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 04:00:10.604065 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d78e8748804374b8af237c1ca8ede8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.604696 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.607290 32585 tablet_service.cc:1467] Processing CreateTablet for tablet d80edf8f3dcd4b64a167796d0c741ecd (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 04:00:10.608503 32585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80edf8f3dcd4b64a167796d0c741ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.613905 32460 tablet_service.cc:1467] Processing CreateTablet for tablet db65aabb33f8427593eac56e96ca9cc6 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 04:00:10.615270 32460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db65aabb33f8427593eac56e96ca9cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.625166 32454 tablet_service.cc:1467] Processing CreateTablet for tablet db915af860484020942e9495eda28804 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 04:00:10.626513 32454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db915af860484020942e9495eda28804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.629251 32453 tablet_service.cc:1467] Processing CreateTablet for tablet 8417acc05db341aca15704cd05736651 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 04:00:10.630592 32453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8417acc05db341aca15704cd05736651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.633169 32452 tablet_service.cc:1467] Processing CreateTablet for tablet bd55a7d68ee84cb7a69c962f984f9372 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 04:00:10.634439 32452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd55a7d68ee84cb7a69c962f984f9372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.637192 32451 tablet_service.cc:1467] Processing CreateTablet for tablet 65c8a38c9f554ef887e7782aa21ad891 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 04:00:10.638475 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c8a38c9f554ef887e7782aa21ad891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.623581 32457 tablet_service.cc:1467] Processing CreateTablet for tablet c271fbb1b29c41b8afbfd003a9c9cb3b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 04:00:10.645191 32736 tablet_service.cc:1467] Processing CreateTablet for tablet fb31133082b14fb889780e53b990e965 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 04:00:10.646507 32736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb31133082b14fb889780e53b990e965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.647768 32460 tablet_service.cc:1467] Processing CreateTablet for tablet d2ad471021a947e9a45d57c7be704453 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 04:00:10.649780 32457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c271fbb1b29c41b8afbfd003a9c9cb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.649544 32734 tablet_service.cc:1467] Processing CreateTablet for tablet 42a553c7b71d493284e5e79b6f19f848 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 04:00:10.650831 32734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a553c7b71d493284e5e79b6f19f848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.650738 32595 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc7e4a68014409c8199e1e003add00a (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 04:00:10.651984 32595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc7e4a68014409c8199e1e003add00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.652443 32460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ad471021a947e9a45d57c7be704453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.653147 32729 tablet_service.cc:1467] Processing CreateTablet for tablet c271fbb1b29c41b8afbfd003a9c9cb3b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 04:00:10.653456 32449 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7858566354435c9b6b09c5b8c1714c (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 04:00:10.654587 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7858566354435c9b6b09c5b8c1714c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.652443 32725 tablet_service.cc:1467] Processing CreateTablet for tablet 8417acc05db341aca15704cd05736651 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 04:00:10.655654 32732 tablet_service.cc:1467] Processing CreateTablet for tablet db65aabb33f8427593eac56e96ca9cc6 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 04:00:10.657370 32723 tablet_service.cc:1467] Processing CreateTablet for tablet bd55a7d68ee84cb7a69c962f984f9372 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 04:00:10.658020 32598 tablet_service.cc:1467] Processing CreateTablet for tablet 6023572bff444c85842d7b3351debf87 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 04:00:10.659207 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6023572bff444c85842d7b3351debf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.660143 32722 tablet_service.cc:1467] Processing CreateTablet for tablet 65c8a38c9f554ef887e7782aa21ad891 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 04:00:10.654421 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c271fbb1b29c41b8afbfd003a9c9cb3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.663192 32725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8417acc05db341aca15704cd05736651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.663610 32733 tablet_service.cc:1467] Processing CreateTablet for tablet bf5ed335f5bb4cf491d1d169c809b5d9 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 04:00:10.663935 32585 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2f839ac9f94c2f980cc895f641607f (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 04:00:10.664394 32732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db65aabb33f8427593eac56e96ca9cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.651284 32727 tablet_service.cc:1467] Processing CreateTablet for tablet db915af860484020942e9495eda28804 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 04:00:10.665143 32585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2f839ac9f94c2f980cc895f641607f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.665865 32727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db915af860484020942e9495eda28804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.667114 32723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd55a7d68ee84cb7a69c962f984f9372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.667515 32721 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7858566354435c9b6b09c5b8c1714c (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 04:00:10.676026 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf5ed335f5bb4cf491d1d169c809b5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.676607 32591 tablet_service.cc:1467] Processing CreateTablet for tablet 5c662c1a70954272bb9eff4a51caaeee (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 04:00:10.677845 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c662c1a70954272bb9eff4a51caaeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.680424 32721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7858566354435c9b6b09c5b8c1714c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.683492 32461 tablet_service.cc:1467] Processing CreateTablet for tablet bec6dcc3e9784bbe83c0ee7452223cb4 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 04:00:10.684109 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c8a38c9f554ef887e7782aa21ad891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.685467 32596 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1c97829f044cfe84bbe23b612c3b81 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 04:00:10.689554 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec6dcc3e9784bbe83c0ee7452223cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.682605 32594 tablet_service.cc:1467] Processing CreateTablet for tablet 4194210f40654ad28af7fc5644e71ace (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 04:00:10.692683 351 tablet_bootstrap.cc:654] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:10.698089 352 tablet_bootstrap.cc:492] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:10.683945 32589 tablet_service.cc:1467] Processing CreateTablet for tablet 8feef3e400bd4fdda571ce5ead436f21 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 04:00:10.704134 32465 tablet_service.cc:1467] Processing CreateTablet for tablet d80edf8f3dcd4b64a167796d0c741ecd (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 04:00:10.705437 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80edf8f3dcd4b64a167796d0c741ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.709925 32462 tablet_service.cc:1467] Processing CreateTablet for tablet 46d78e8748804374b8af237c1ca8ede8 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 04:00:10.711128 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d78e8748804374b8af237c1ca8ede8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.711292 32451 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 04:00:10.724254 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.724444 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1c97829f044cfe84bbe23b612c3b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.728379 32594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4194210f40654ad28af7fc5644e71ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.730379 32589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feef3e400bd4fdda571ce5ead436f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.760334 352 tablet_bootstrap.cc:654] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:10.774137 32587 tablet_service.cc:1467] Processing CreateTablet for tablet 725b0e6610ef466d8d7cf9ed945d2d4e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 04:00:10.774137 32600 tablet_service.cc:1467] Processing CreateTablet for tablet e78ed11baa324b6bbbf7a965a2bad24e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 04:00:10.777084 32600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78ed11baa324b6bbbf7a965a2bad24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.777720 32461 tablet_service.cc:1467] Processing CreateTablet for tablet 8feef3e400bd4fdda571ce5ead436f21 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 04:00:10.778932 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feef3e400bd4fdda571ce5ead436f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.781190 32457 tablet_service.cc:1467] Processing CreateTablet for tablet 6023572bff444c85842d7b3351debf87 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 04:00:10.781311 32462 tablet_service.cc:1467] Processing CreateTablet for tablet 4194210f40654ad28af7fc5644e71ace (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 04:00:10.782461 32587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b0e6610ef466d8d7cf9ed945d2d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.786983 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4194210f40654ad28af7fc5644e71ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.782878 351 tablet_bootstrap.cc:492] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:10.787542 351 ts_tablet_manager.cc:1397] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.190s user 0.010s sys 0.005s
I20260806 04:00:10.779544 32595 tablet_service.cc:1467] Processing CreateTablet for tablet e4533f141e644c719df87540a1ca54e8 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 04:00:10.788822 32595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4533f141e644c719df87540a1ca54e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.791234 351 raft_consensus.cc:348] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:10.792693 32457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6023572bff444c85842d7b3351debf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.798141 32733 tablet_service.cc:1467] Processing CreateTablet for tablet d2ad471021a947e9a45d57c7be704453 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 04:00:10.805454 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ad471021a947e9a45d57c7be704453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.808890 32592 tablet_service.cc:1467] Processing CreateTablet for tablet e5a23c953ef64542931fb8ede0fcbb97 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 04:00:10.810089 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a23c953ef64542931fb8ede0fcbb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.817157 32727 tablet_service.cc:1467] Processing CreateTablet for tablet bec6dcc3e9784bbe83c0ee7452223cb4 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 04:00:10.818475 32727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec6dcc3e9784bbe83c0ee7452223cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.825282 32591 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6a30af60042f9b0a84eb0fdb8db19 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 04:00:10.829422 32453 tablet_service.cc:1467] Processing CreateTablet for tablet 5c662c1a70954272bb9eff4a51caaeee (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 04:00:10.830749 32585 tablet_service.cc:1467] Processing CreateTablet for tablet 18f3a6d6cef24470a69dbafa39b3d3e1 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 04:00:10.831300 32453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c662c1a70954272bb9eff4a51caaeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.834470 32721 tablet_service.cc:1467] Processing CreateTablet for tablet 46d78e8748804374b8af237c1ca8ede8 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 04:00:10.835700 32721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d78e8748804374b8af237c1ca8ede8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.840116 32729 tablet_service.cc:1467] Processing CreateTablet for tablet d80edf8f3dcd4b64a167796d0c741ecd (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 04:00:10.841452 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80edf8f3dcd4b64a167796d0c741ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.831972 32585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f3a6d6cef24470a69dbafa39b3d3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.846517 32734 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 04:00:10.847139 32598 tablet_service.cc:1467] Processing CreateTablet for tablet 0201a31e3c594fdfa1cec8a9970becef (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 04:00:10.847527 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6a30af60042f9b0a84eb0fdb8db19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.801792 351 raft_consensus.cc:374] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:10.854888 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0201a31e3c594fdfa1cec8a9970becef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.856072 32452 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2f839ac9f94c2f980cc895f641607f (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 04:00:10.856772 32732 tablet_service.cc:1467] Processing CreateTablet for tablet 8feef3e400bd4fdda571ce5ead436f21 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 04:00:10.857357 32452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2f839ac9f94c2f980cc895f641607f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.858732 357 tablet_bootstrap.cc:492] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:10.860275 32457 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1c97829f044cfe84bbe23b612c3b81 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 04:00:10.861625 32457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1c97829f044cfe84bbe23b612c3b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.850106 351 raft_consensus.cc:729] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:10.864535 351 consensus_queue.cc:260] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:10.868626 32600 tablet_service.cc:1467] Processing CreateTablet for tablet fe6ca7a29c2e4741a8285425bb994584 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 04:00:10.875988 32460 tablet_service.cc:1467] Processing CreateTablet for tablet 725b0e6610ef466d8d7cf9ed945d2d4e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 04:00:10.877460 32460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b0e6610ef466d8d7cf9ed945d2d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.880514 32454 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc7e4a68014409c8199e1e003add00a (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 04:00:10.881912 32454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc7e4a68014409c8199e1e003add00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.881985 32660 heartbeater.cc:503] Master 127.29.237.190:40089 was elected leader, sending a full tablet report...
I20260806 04:00:10.883122 32600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6ca7a29c2e4741a8285425bb994584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.885375 32725 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc7e4a68014409c8199e1e003add00a (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 04:00:10.887671 32451 tablet_service.cc:1467] Processing CreateTablet for tablet e78ed11baa324b6bbbf7a965a2bad24e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 04:00:10.888123 32734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.888949 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78ed11baa324b6bbbf7a965a2bad24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.889194 32736 tablet_service.cc:1467] Processing CreateTablet for tablet 4194210f40654ad28af7fc5644e71ace (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 04:00:10.892153 32722 tablet_service.cc:1467] Processing CreateTablet for tablet 5c662c1a70954272bb9eff4a51caaeee (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
W20260806 04:00:10.893361 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:10.893133 32594 tablet_service.cc:1467] Processing CreateTablet for tablet 187160a8bf734a23b90fcc49bc24b10b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 04:00:10.894368 32594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187160a8bf734a23b90fcc49bc24b10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.894870 352 tablet_bootstrap.cc:492] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:10.895320 352 ts_tablet_manager.cc:1397] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.198s user 0.011s sys 0.013s
I20260806 04:00:10.896168 32723 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2f839ac9f94c2f980cc895f641607f (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 04:00:10.898002 352 raft_consensus.cc:348] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:10.898756 352 raft_consensus.cc:374] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:10.898382 32589 tablet_service.cc:1467] Processing CreateTablet for tablet c4f865676f884755987b8bd820489186 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 04:00:10.899060 352 raft_consensus.cc:729] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:10.890350 32736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4194210f40654ad28af7fc5644e71ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.900424 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c662c1a70954272bb9eff4a51caaeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.901474 351 ts_tablet_manager.cc:1428] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.114s user 0.008s sys 0.000s
I20260806 04:00:10.902536 351 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:10.903205 32725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc7e4a68014409c8199e1e003add00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.904364 32732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feef3e400bd4fdda571ce5ead436f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.905787 32596 tablet_service.cc:1467] Processing CreateTablet for tablet 23ef18327cc345d08d82dbc954eaa969 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
W20260806 04:00:10.910476 32661 tablet.cc:2369] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:00:10.913686 32457 tablet_service.cc:1467] Processing CreateTablet for tablet e5a23c953ef64542931fb8ede0fcbb97 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 04:00:10.914974 32457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a23c953ef64542931fb8ede0fcbb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.894060 32465 tablet_service.cc:1467] Processing CreateTablet for tablet e4533f141e644c719df87540a1ca54e8 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 04:00:10.916821 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4533f141e644c719df87540a1ca54e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.916720 32733 tablet_service.cc:1467] Processing CreateTablet for tablet 6023572bff444c85842d7b3351debf87 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 04:00:10.918068 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6023572bff444c85842d7b3351debf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.900045 32589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f865676f884755987b8bd820489186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.924501 352 consensus_queue.cc:260] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:10.926085 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ef18327cc345d08d82dbc954eaa969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.932834 32723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2f839ac9f94c2f980cc895f641607f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.944859 357 tablet_bootstrap.cc:654] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:10.953425 32462 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6a30af60042f9b0a84eb0fdb8db19 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 04:00:10.954030 32453 tablet_service.cc:1467] Processing CreateTablet for tablet 18f3a6d6cef24470a69dbafa39b3d3e1 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 04:00:10.954751 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6a30af60042f9b0a84eb0fdb8db19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.958279 32449 tablet_service.cc:1467] Processing CreateTablet for tablet e30158f7dfb0455780b91c51e03665bc (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 04:00:10.958910 32453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f3a6d6cef24470a69dbafa39b3d3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.960140 32727 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1c97829f044cfe84bbe23b612c3b81 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 04:00:10.961391 32727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1c97829f044cfe84bbe23b612c3b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.962397 32736 tablet_service.cc:1467] Processing CreateTablet for tablet 725b0e6610ef466d8d7cf9ed945d2d4e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 04:00:10.963630 32736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b0e6610ef466d8d7cf9ed945d2d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.965423 32722 tablet_service.cc:1467] Processing CreateTablet for tablet e78ed11baa324b6bbbf7a965a2bad24e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 04:00:10.973694 351 tablet_bootstrap.cc:654] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:10.981874 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30158f7dfb0455780b91c51e03665bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.990253 32598 tablet_service.cc:1467] Processing CreateTablet for tablet a12b9e678e6049be86bca4e981f90379 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 04:00:10.990607 32595 tablet_service.cc:1467] Processing CreateTablet for tablet d7172fff185b4788acb609d2314e4b4b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 04:00:10.991474 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12b9e678e6049be86bca4e981f90379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.996423 32595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7172fff185b4788acb609d2314e4b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:10.998597 32592 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9b09ac79f84fb797c2a263c612eee5 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 04:00:10.999964 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9b09ac79f84fb797c2a263c612eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.011194 32721 tablet_service.cc:1467] Processing CreateTablet for tablet e4533f141e644c719df87540a1ca54e8 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 04:00:11.016456 32587 tablet_service.cc:1467] Processing CreateTablet for tablet d20f5ce3f2ac40379306e3d630f63eda (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 04:00:11.004792 352 ts_tablet_manager.cc:1428] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.109s user 0.011s sys 0.005s
I20260806 04:00:11.028144 32587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20f5ce3f2ac40379306e3d630f63eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.029412 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78ed11baa324b6bbbf7a965a2bad24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.033631 32532 heartbeater.cc:503] Master 127.29.237.190:40089 was elected leader, sending a full tablet report...
I20260806 04:00:11.043835 32461 tablet_service.cc:1467] Processing CreateTablet for tablet 0201a31e3c594fdfa1cec8a9970becef (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 04:00:11.045295 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0201a31e3c594fdfa1cec8a9970becef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.045818 32600 tablet_service.cc:1467] Processing CreateTablet for tablet 2f39a49c08914cb0a772f3187e29fe61 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 04:00:11.047009 32600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f39a49c08914cb0a772f3187e29fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.050817 32585 tablet_service.cc:1467] Processing CreateTablet for tablet f9fdfcc3345c4c86a7d974030c67c271 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 04:00:11.052578 32591 tablet_service.cc:1467] Processing CreateTablet for tablet 57c0e905630e43668379f2f95684775f (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 04:00:11.055127 32725 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6a30af60042f9b0a84eb0fdb8db19 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 04:00:11.056170 352 tablet_bootstrap.cc:492] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:11.060464 32734 tablet_service.cc:1467] Processing CreateTablet for tablet 18f3a6d6cef24470a69dbafa39b3d3e1 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 04:00:11.064208 32454 tablet_service.cc:1467] Processing CreateTablet for tablet c4f865676f884755987b8bd820489186 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 04:00:11.065481 32454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f865676f884755987b8bd820489186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.068216 32721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4533f141e644c719df87540a1ca54e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.070428 32594 tablet_service.cc:1467] Processing CreateTablet for tablet 128baf4b99ba44f0aa14ba803db9e1ba (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 04:00:11.074723 32451 tablet_service.cc:1467] Processing CreateTablet for tablet 23ef18327cc345d08d82dbc954eaa969 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 04:00:11.075611 32589 tablet_service.cc:1467] Processing CreateTablet for tablet d7c0dfae3c464cd18cfd95401b49eed9 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 04:00:11.076716 32452 tablet_service.cc:1467] Processing CreateTablet for tablet 187160a8bf734a23b90fcc49bc24b10b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 04:00:11.078877 32465 tablet_service.cc:1467] Processing CreateTablet for tablet a12b9e678e6049be86bca4e981f90379 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 04:00:11.079627 32585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fdfcc3345c4c86a7d974030c67c271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.080022 32733 tablet_service.cc:1467] Processing CreateTablet for tablet fe6ca7a29c2e4741a8285425bb994584 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 04:00:11.082231 32596 tablet_service.cc:1467] Processing CreateTablet for tablet ab1948a6fbe64767a22598e910340a6b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 04:00:11.083537 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1948a6fbe64767a22598e910340a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.088451 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c0e905630e43668379f2f95684775f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.092028 352 tablet_bootstrap.cc:654] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:11.096400 357 tablet_bootstrap.cc:492] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:11.096868 357 ts_tablet_manager.cc:1397] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.238s user 0.007s sys 0.009s
I20260806 04:00:11.098747 32725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6a30af60042f9b0a84eb0fdb8db19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.076052 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ef18327cc345d08d82dbc954eaa969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.099730 357 raft_consensus.cc:348] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.100713 32734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f3a6d6cef24470a69dbafa39b3d3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.118139 32460 tablet_service.cc:1467] Processing CreateTablet for tablet f9fdfcc3345c4c86a7d974030c67c271 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 04:00:11.100517 357 raft_consensus.cc:374] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.112435 32594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128baf4b99ba44f0aa14ba803db9e1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.121414 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6ca7a29c2e4741a8285425bb994584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.102718 32452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187160a8bf734a23b90fcc49bc24b10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.122583 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12b9e678e6049be86bca4e981f90379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.101310 32729 tablet_service.cc:1467] Processing CreateTablet for tablet e5a23c953ef64542931fb8ede0fcbb97 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 04:00:11.125041 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a23c953ef64542931fb8ede0fcbb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.120812 357 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.113531 32453 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9b09ac79f84fb797c2a263c612eee5 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 04:00:11.161172 357 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.146763 32721 tablet_service.cc:1467] Processing CreateTablet for tablet c4f865676f884755987b8bd820489186 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 04:00:11.135519 32457 tablet_service.cc:1467] Processing CreateTablet for tablet fe6ca7a29c2e4741a8285425bb994584 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 04:00:11.149025 352 tablet_bootstrap.cc:492] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:11.148890 32723 tablet_service.cc:1467] Processing CreateTablet for tablet 0201a31e3c594fdfa1cec8a9970becef (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 04:00:11.149374 32722 tablet_service.cc:1467] Processing CreateTablet for tablet 23ef18327cc345d08d82dbc954eaa969 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 04:00:11.165988 352 ts_tablet_manager.cc:1397] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.001s
I20260806 04:00:11.148382 32460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fdfcc3345c4c86a7d974030c67c271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.157109 32592 tablet_service.cc:1467] Processing CreateTablet for tablet 77befd7e7cea42ed8a469471b80a08f7 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 04:00:11.157408 32591 tablet_service.cc:1467] Processing CreateTablet for tablet 12e7971e809846ada28ebf1d6cea4837 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 04:00:11.168381 359 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:11.168874 359 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.171299 359 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:11.159044 32734 tablet_service.cc:1467] Processing CreateTablet for tablet a12b9e678e6049be86bca4e981f90379 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 04:00:11.090191 32732 tablet_service.cc:1467] Processing CreateTablet for tablet 187160a8bf734a23b90fcc49bc24b10b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 04:00:11.160192 32595 tablet_service.cc:1467] Processing CreateTablet for tablet e30158f7dfb0455780b91c51e03665bc (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 04:00:11.163969 32721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f865676f884755987b8bd820489186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.167133 32453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9b09ac79f84fb797c2a263c612eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.169423 352 raft_consensus.cc:348] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.183658 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ef18327cc345d08d82dbc954eaa969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.183988 352 raft_consensus.cc:374] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.184327 352 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.185451 32732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187160a8bf734a23b90fcc49bc24b10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.185094 352 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.186633 32723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0201a31e3c594fdfa1cec8a9970becef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.176337 32457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6ca7a29c2e4741a8285425bb994584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.162706 32589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c0dfae3c464cd18cfd95401b49eed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.111519 32449 tablet_service.cc:1467] Processing CreateTablet for tablet d7172fff185b4788acb609d2314e4b4b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 04:00:11.189332 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7172fff185b4788acb609d2314e4b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.189874 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e7971e809846ada28ebf1d6cea4837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.190963 32734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12b9e678e6049be86bca4e981f90379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.178093 351 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:11.194733 351 ts_tablet_manager.cc:1397] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.292s user 0.010s sys 0.035s
I20260806 04:00:11.196583 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77befd7e7cea42ed8a469471b80a08f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.200040 32727 tablet_service.cc:1467] Processing CreateTablet for tablet d7172fff185b4788acb609d2314e4b4b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 04:00:11.201354 32727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7172fff185b4788acb609d2314e4b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.203148 32462 tablet_service.cc:1467] Processing CreateTablet for tablet 57c0e905630e43668379f2f95684775f (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 04:00:11.203260 32595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30158f7dfb0455780b91c51e03665bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.208829 32587 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec5d99c8efa405b84db49741c2f8933 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 04:00:11.210325 32587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec5d99c8efa405b84db49741c2f8933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.232930 32461 tablet_service.cc:1467] Processing CreateTablet for tablet d20f5ce3f2ac40379306e3d630f63eda (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 04:00:11.246708 32585 tablet_service.cc:1467] Processing CreateTablet for tablet 45b7481674ca403b9a224047a6d99b03 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 04:00:11.247383 32596 tablet_service.cc:1467] Processing CreateTablet for tablet ec440a6dfdfc46af8a0876b464ef531d (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 04:00:11.269018 351 raft_consensus.cc:348] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.269934 351 raft_consensus.cc:374] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.270279 351 raft_consensus.cc:729] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.252173 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c0e905630e43668379f2f95684775f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.276707 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20f5ce3f2ac40379306e3d630f63eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.252888 357 ts_tablet_manager.cc:1428] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.156s user 0.011s sys 0.009s
I20260806 04:00:11.278191 327 heartbeater.cc:503] Master 127.29.237.190:40089 was elected leader, sending a full tablet report...
I20260806 04:00:11.278430 357 tablet_bootstrap.cc:492] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:11.256361 352 ts_tablet_manager.cc:1428] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.090s user 0.010s sys 0.000s
I20260806 04:00:11.281049 352 tablet_bootstrap.cc:492] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:11.280834 32729 tablet_service.cc:1467] Processing CreateTablet for tablet f9fdfcc3345c4c86a7d974030c67c271 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 04:00:11.282061 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fdfcc3345c4c86a7d974030c67c271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.287668 32591 tablet_service.cc:1467] Processing CreateTablet for tablet 62779f9120ed4ef3b0d9febc7c2cd224 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 04:00:11.291357 32592 tablet_service.cc:1467] Processing CreateTablet for tablet ff861725fdbf4efda0de572d435f5d6d (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 04:00:11.301892 32454 tablet_service.cc:1467] Processing CreateTablet for tablet 2f39a49c08914cb0a772f3187e29fe61 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 04:00:11.303172 32454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f39a49c08914cb0a772f3187e29fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.304847 32733 tablet_service.cc:1467] Processing CreateTablet for tablet 57c0e905630e43668379f2f95684775f (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 04:00:11.309655 32727 tablet_service.cc:1467] Processing CreateTablet for tablet d20f5ce3f2ac40379306e3d630f63eda (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 04:00:11.248050 32585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b7481674ca403b9a224047a6d99b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.313107 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff861725fdbf4efda0de572d435f5d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.315593 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec440a6dfdfc46af8a0876b464ef531d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.316409 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c0e905630e43668379f2f95684775f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.320390 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62779f9120ed4ef3b0d9febc7c2cd224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.271518 351 consensus_queue.cc:260] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.322314 357 tablet_bootstrap.cc:654] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:11.322945 351 ts_tablet_manager.cc:1428] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.128s user 0.006s sys 0.002s
I20260806 04:00:11.324108 351 tablet_bootstrap.cc:492] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:11.326689 32727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20f5ce3f2ac40379306e3d630f63eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.328871 32736 tablet_service.cc:1467] Processing CreateTablet for tablet 2f39a49c08914cb0a772f3187e29fe61 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 04:00:11.330148 32736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f39a49c08914cb0a772f3187e29fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.349783 32594 tablet_service.cc:1467] Processing CreateTablet for tablet 9befcec6c17046459b03fa9ce98de94e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 04:00:11.351167 32594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9befcec6c17046459b03fa9ce98de94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.352914 32589 tablet_service.cc:1467] Processing CreateTablet for tablet c2611a58e8424bd088918c8aa63add87 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 04:00:11.346212 352 tablet_bootstrap.cc:654] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:11.358933 32598 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e254fda3f4e30802dcd0bba7a82ca (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 04:00:11.362234 351 tablet_bootstrap.cc:654] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:11.365234 32465 tablet_service.cc:1467] Processing CreateTablet for tablet ab1948a6fbe64767a22598e910340a6b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 04:00:11.366501 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1948a6fbe64767a22598e910340a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.368368 360 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:11.368801 360 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.369043 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:00:11.370743 32553 leader_election.cc:341] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.414078 32722 tablet_service.cc:1467] Processing CreateTablet for tablet ab1948a6fbe64767a22598e910340a6b (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 04:00:11.421818 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1948a6fbe64767a22598e910340a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.425859 32451 tablet_service.cc:1467] Processing CreateTablet for tablet 128baf4b99ba44f0aa14ba803db9e1ba (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 04:00:11.427181 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128baf4b99ba44f0aa14ba803db9e1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.430521 32721 tablet_service.cc:1467] Processing CreateTablet for tablet 128baf4b99ba44f0aa14ba803db9e1ba (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 04:00:11.431754 32721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128baf4b99ba44f0aa14ba803db9e1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.433714 32452 tablet_service.cc:1467] Processing CreateTablet for tablet d7c0dfae3c464cd18cfd95401b49eed9 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 04:00:11.434903 32452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c0dfae3c464cd18cfd95401b49eed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.438735 32734 tablet_service.cc:1467] Processing CreateTablet for tablet d7c0dfae3c464cd18cfd95401b49eed9 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 04:00:11.439951 32734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c0dfae3c464cd18cfd95401b49eed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.439072 32449 tablet_service.cc:1467] Processing CreateTablet for tablet 77befd7e7cea42ed8a469471b80a08f7 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 04:00:11.399467 32725 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9b09ac79f84fb797c2a263c612eee5 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 04:00:11.452595 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77befd7e7cea42ed8a469471b80a08f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.452936 32725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9b09ac79f84fb797c2a263c612eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.413216 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1e254fda3f4e30802dcd0bba7a82ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.457216 32589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2611a58e8424bd088918c8aa63add87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.477353 359 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:11.488626 380 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:11.489190 380 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.491689 380 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:11.481586 359 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.487552 360 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:11.525535 32462 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec5d99c8efa405b84db49741c2f8933 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 04:00:11.527032 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec5d99c8efa405b84db49741c2f8933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.541538 359 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:11.542299 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:11.561445 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:11.561861 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:11.562804 32461 tablet_service.cc:1467] Processing CreateTablet for tablet 45b7481674ca403b9a224047a6d99b03 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20260806 04:00:11.563452 32553 leader_election.cc:341] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.563937 32553 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:11.564318 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b7481674ca403b9a224047a6d99b03. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 04:00:11.565343 32553 leader_election.cc:341] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:11.566067 32553 leader_election.cc:341] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.566427 32553 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:11.567384 359 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:11.567991 359 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:11.561427 32727 tablet_service.cc:1467] Processing CreateTablet for tablet 77befd7e7cea42ed8a469471b80a08f7 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 04:00:11.569422 357 tablet_bootstrap.cc:492] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:11.569861 357 ts_tablet_manager.cc:1397] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.292s user 0.009s sys 0.006s
I20260806 04:00:11.572520 357 raft_consensus.cc:348] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.573231 357 raft_consensus.cc:374] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.573501 357 raft_consensus.cc:729] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.574918 357 consensus_queue.cc:260] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.576598 32453 tablet_service.cc:1467] Processing CreateTablet for tablet 12e7971e809846ada28ebf1d6cea4837 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 04:00:11.577250 357 ts_tablet_manager.cc:1428] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:00:11.577859 32453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e7971e809846ada28ebf1d6cea4837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.578151 357 tablet_bootstrap.cc:492] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:11.596627 32600 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 39) took 1010 ms. Trace:
I20260806 04:00:11.600179 32596 tablet_service.cc:1467] Processing CreateTablet for tablet 305cbd07ce124048b7ac2f99b2aa40db (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 04:00:11.601528 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 305cbd07ce124048b7ac2f99b2aa40db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.567488 32591 tablet_service.cc:1467] Processing CreateTablet for tablet 11b58a6c0e4e409fb675ad1e01ef8b60 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 04:00:11.589898 32727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77befd7e7cea42ed8a469471b80a08f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.597054 32600 rpcz_store.cc:276] 0806 04:00:10.585773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.030206 (+444433us) service_pool.cc:227] Handling call
0806 04:00:11.046580 (+ 16374us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.046672 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.242683 (+196011us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.593539 (+350856us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.596560 (+ 3021us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.608728 32587 tablet_service.cc:1467] Processing CreateTablet for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 04:00:11.618165 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b58a6c0e4e409fb675ad1e01ef8b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.623193 371 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:11.623015 32453 tablet_service.cc:1467] Processing CreateTablet for tablet 305cbd07ce124048b7ac2f99b2aa40db (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 04:00:11.623657 371 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.627169 32723 tablet_service.cc:1467] Processing CreateTablet for tablet e30158f7dfb0455780b91c51e03665bc (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 04:00:11.627763 32600 tablet_service.cc:1467] Processing CreateTablet for tablet e870f9de1d6a4db4bbf149c6a785135c (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 04:00:11.628432 371 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:11.632156 32723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30158f7dfb0455780b91c51e03665bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.644070 352 tablet_bootstrap.cc:492] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:11.644631 352 ts_tablet_manager.cc:1397] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.364s user 0.045s sys 0.040s
I20260806 04:00:11.612577 32733 tablet_service.cc:1467] Processing CreateTablet for tablet 12e7971e809846ada28ebf1d6cea4837 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
W20260806 04:00:11.614754 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 48710.9us
I20260806 04:00:11.652174 32453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 305cbd07ce124048b7ac2f99b2aa40db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.660780 32732 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec5d99c8efa405b84db49741c2f8933 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 04:00:11.663383 32457 tablet_service.cc:1467] Processing CreateTablet for tablet ff861725fdbf4efda0de572d435f5d6d (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 04:00:11.661844 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e7971e809846ada28ebf1d6cea4837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.664966 32732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec5d99c8efa405b84db49741c2f8933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.665791 351 tablet_bootstrap.cc:492] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:11.666229 351 ts_tablet_manager.cc:1397] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.342s user 0.009s sys 0.013s
I20260806 04:00:11.668964 351 raft_consensus.cc:348] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.669698 351 raft_consensus.cc:374] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.647583 352 raft_consensus.cc:348] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.676573 352 raft_consensus.cc:374] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.676968 352 raft_consensus.cc:729] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.677821 352 consensus_queue.cc:260] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.680655 32460 tablet_service.cc:1467] Processing CreateTablet for tablet 62779f9120ed4ef3b0d9febc7c2cd224 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 04:00:11.681470 32462 tablet_service.cc:1467] Processing CreateTablet for tablet c2611a58e8424bd088918c8aa63add87 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 04:00:11.669973 351 raft_consensus.cc:729] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.696851 351 consensus_queue.cc:260] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.698275 32594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 52) took 1004 ms. Trace:
I20260806 04:00:11.701012 32722 tablet_service.cc:1467] Processing CreateTablet for tablet ec440a6dfdfc46af8a0876b464ef531d (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 04:00:11.701607 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:11.702230 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec440a6dfdfc46af8a0876b464ef531d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.702316 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:11.702936 32748 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:11.699025 351 ts_tablet_manager.cc:1428] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.033s user 0.005s sys 0.002s
I20260806 04:00:11.705037 32451 tablet_service.cc:1467] Processing CreateTablet for tablet 9befcec6c17046459b03fa9ce98de94e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20260806 04:00:11.704056 32428 leader_election.cc:341] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.706238 32428 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:11.706995 360 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:11.707331 360 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:11.686260 32729 tablet_service.cc:1467] Processing CreateTablet for tablet 45b7481674ca403b9a224047a6d99b03 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 04:00:11.704974 351 tablet_bootstrap.cc:492] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:11.707651 360 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:11.654538 32587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.658625 32461 tablet_service.cc:1467] Processing CreateTablet for tablet ec440a6dfdfc46af8a0876b464ef531d (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 04:00:11.717167 32585 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 47) took 1043 ms. Trace:
I20260806 04:00:11.717355 352 ts_tablet_manager.cc:1428] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.072s user 0.004s sys 0.003s
I20260806 04:00:11.708987 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b7481674ca403b9a224047a6d99b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.717583 32585 rpcz_store.cc:276] 0806 04:00:10.673450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.245788 (+572338us) service_pool.cc:227] Handling call
0806 04:00:11.247575 (+ 1787us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.247671 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.519170 (+271499us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.716645 (+197475us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.717109 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.719741 32734 tablet_service.cc:1467] Processing CreateTablet for tablet 62779f9120ed4ef3b0d9febc7c2cd224 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 04:00:11.721112 32734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62779f9120ed4ef3b0d9febc7c2cd224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.684697 32592 tablet_service.cc:1467] Processing CreateTablet for tablet d24bcc94fb734c95849f4cb89c15ca8e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 04:00:11.724592 32600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e870f9de1d6a4db4bbf149c6a785135c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.726150 32595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 45) took 1053 ms. Trace:
I20260806 04:00:11.726516 32595 rpcz_store.cc:276] 0806 04:00:10.672807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.159554 (+486747us) service_pool.cc:227] Handling call
0806 04:00:11.202795 (+ 43241us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.202900 (+ 105us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.425998 (+223098us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.725551 (+299553us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.725951 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.727895 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24bcc94fb734c95849f4cb89c15ca8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.715849 360 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.730051 391 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:11.730504 391 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.732424 391 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:11.733922 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:11.734733 32725 tablet_service.cc:1467] Processing CreateTablet for tablet ff861725fdbf4efda0de572d435f5d6d (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
W20260806 04:00:11.735636 32553 leader_election.cc:341] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.735787 352 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:11.737555 32452 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 42) took 1014 ms. Trace:
I20260806 04:00:11.737921 32452 rpcz_store.cc:276] 0806 04:00:10.723129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.432871 (+709742us) service_pool.cc:227] Handling call
0806 04:00:11.434476 (+ 1605us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.434566 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.579237 (+144671us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.728072 (+148835us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.737495 (+ 9423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.742430 32596 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 55) took 1005 ms. Trace:
I20260806 04:00:11.742800 32596 rpcz_store.cc:276] 0806 04:00:10.736696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.583353 (+846657us) service_pool.cc:227] Handling call
0806 04:00:11.601032 (+ 17679us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.601136 (+ 104us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.667744 (+ 66608us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.736581 (+ 68837us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.742370 (+ 5789us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.698638 32594 rpcz_store.cc:276] 0806 04:00:10.694612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.311481 (+616869us) service_pool.cc:227] Handling call
0806 04:00:11.350676 (+ 39195us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.350765 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.639186 (+288421us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.697757 (+ 58571us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.698217 (+ 460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.746842 360 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:11.747488 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:11.748045 32748 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:11.736020 32725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff861725fdbf4efda0de572d435f5d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.749740 32727 tablet_service.cc:1467] Processing CreateTablet for tablet c2611a58e8424bd088918c8aa63add87 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 04:00:11.742367 32457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff861725fdbf4efda0de572d435f5d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.768338 32451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9befcec6c17046459b03fa9ce98de94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.731993 32465 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 40) took 1009 ms. Trace:
I20260806 04:00:11.768678 32465 rpcz_store.cc:276] 0806 04:00:10.722471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.364411 (+641940us) service_pool.cc:227] Handling call
0806 04:00:11.366053 (+ 1642us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.366146 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.587458 (+221312us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.720426 (+132968us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.731934 (+ 11508us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.770447 32465 tablet_service.cc:1467] Processing CreateTablet for tablet 11b58a6c0e4e409fb675ad1e01ef8b60 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 04:00:11.770977 32736 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e254fda3f4e30802dcd0bba7a82ca (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 04:00:11.772151 32736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1e254fda3f4e30802dcd0bba7a82ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.772372 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec440a6dfdfc46af8a0876b464ef531d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.774291 32460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62779f9120ed4ef3b0d9febc7c2cd224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.774823 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b58a6c0e4e409fb675ad1e01ef8b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.775360 32598 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 53) took 1058 ms. Trace:
I20260806 04:00:11.775709 32598 rpcz_store.cc:276] 0806 04:00:10.719055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.358043 (+638988us) service_pool.cc:227] Handling call
0806 04:00:11.359737 (+ 1694us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.359834 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.647186 (+287352us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.727813 (+ 80627us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.775300 (+ 47487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.776508 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:11.754179 357 tablet_bootstrap.cc:654] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:11.777438 351 tablet_bootstrap.cc:654] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
W20260806 04:00:11.778584 32682 leader_election.cc:341] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.741241 32452 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e254fda3f4e30802dcd0bba7a82ca (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 04:00:11.781268 32732 tablet_service.cc:1467] Processing CreateTablet for tablet 11b58a6c0e4e409fb675ad1e01ef8b60 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 04:00:11.782510 32732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b58a6c0e4e409fb675ad1e01ef8b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.755061 32748 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:11.756302 32449 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 43) took 1032 ms. Trace:
I20260806 04:00:11.757475 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:11.786053 32449 rpcz_store.cc:276] 0806 04:00:10.723444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.438289 (+714845us) service_pool.cc:227] Handling call
0806 04:00:11.451431 (+ 13142us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.451844 (+ 413us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.623052 (+171208us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.755796 (+132744us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.756244 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.758100 32454 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 39) took 1036 ms. Trace:
I20260806 04:00:11.758741 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:11.760172 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:11.787518 32428 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:11.760649 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:11.787760 32449 tablet_service.cc:1467] Processing CreateTablet for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 04:00:11.788121 32454 rpcz_store.cc:276] 0806 04:00:10.722152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.301085 (+578933us) service_pool.cc:227] Handling call
0806 04:00:11.302720 (+ 1635us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.302811 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.596947 (+294136us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.757526 (+160579us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.758046 (+ 520us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:00:11.789232 32553 leader_election.cc:341] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.789641 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:11.789975 32727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2611a58e8424bd088918c8aa63add87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.762096 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:11.790467 391 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:11.762266 32589 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 51) took 1077 ms. Trace:
I20260806 04:00:11.762596 371 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:11.762941 32591 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 54) took 1043 ms. Trace:
I20260806 04:00:11.765625 32721 tablet_service.cc:1467] Processing CreateTablet for tablet 9befcec6c17046459b03fa9ce98de94e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20260806 04:00:11.786958 32427 leader_election.cc:341] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.791049 32478 raft_consensus.cc:2379] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:11.808429 371 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:11.809443 32427 leader_election.cc:341] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.809966 32427 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:11.810788 380 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:11.809487 32721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9befcec6c17046459b03fa9ce98de94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.810999 371 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:11.819545 32723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 45) took 1034 ms. Trace:
I20260806 04:00:11.813110 32683 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:11.813758 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:11.820011 32723 rpcz_store.cc:276] 0806 04:00:10.785557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.625554 (+839997us) service_pool.cc:227] Handling call
0806 04:00:11.627967 (+ 2413us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.628061 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.741834 (+113773us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.819024 (+ 77190us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.819473 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.812186 360 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 04:00:11.821300 32682 leader_election.cc:341] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.814066 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:11.822119 32478 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:11.807649 32589 rpcz_store.cc:276] 0806 04:00:10.684696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.300075 (+615379us) service_pool.cc:227] Handling call
0806 04:00:11.353778 (+ 53703us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.353873 (+ 95us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.706781 (+352908us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.761823 (+ 55042us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.762211 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.823078 32683 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:11.823263 360 raft_consensus.cc:686] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:11.796520 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2611a58e8424bd088918c8aa63add87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.814713 32592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 58) took 1077 ms. Trace:
I20260806 04:00:11.825052 32592 rpcz_store.cc:276] 0806 04:00:10.737719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.683694 (+945975us) service_pool.cc:227] Handling call
0806 04:00:11.724382 (+ 40688us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.724479 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.785764 (+ 61285us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.814175 (+ 28411us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.814649 (+ 474us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.807523 32591 rpcz_store.cc:276] 0806 04:00:10.719447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.566656 (+847209us) service_pool.cc:227] Handling call
0806 04:00:11.617623 (+ 50967us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b58001e5798. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4e82 00007fd17320d8b9 00007fd1721205d8 00007fd1721e02a0 00007fd172191869 00007fd16b793c44 00007fd16b793bd2 00007fd16b793b01 00007fd16b793a7c
0806 04:00:11.617686 (+ 63us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.617783 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.667072 (+ 49289us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.751948 (+ 84876us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.762886 (+ 10938us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4336896}
I20260806 04:00:11.790114 32454 tablet_service.cc:1467] Processing CreateTablet for tablet e870f9de1d6a4db4bbf149c6a785135c (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 04:00:11.835189 32454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e870f9de1d6a4db4bbf149c6a785135c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.836386 32452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1e254fda3f4e30802dcd0bba7a82ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.838446 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.844385 360 consensus_queue.cc:237] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.825569 371 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:11.860628 32723 tablet_service.cc:1467] Processing CreateTablet for tablet 305cbd07ce124048b7ac2f99b2aa40db (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 04:00:11.862066 32723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 305cbd07ce124048b7ac2f99b2aa40db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.865067 352 tablet_bootstrap.cc:654] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:11.870702 32600 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 57) took 1133 ms. Trace:
I20260806 04:00:11.871090 32600 rpcz_store.cc:276] 0806 04:00:10.737407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.626938 (+889531us) service_pool.cc:227] Handling call
0806 04:00:11.637601 (+ 10663us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.642477 (+ 4876us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.829076 (+186599us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.870136 (+ 41060us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.870636 (+ 500us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.872179 32465 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 54) took 1076 ms. Trace:
W20260806 04:00:11.873000 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 50019.4us
I20260806 04:00:11.873683 32729 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 47) took 1087 ms. Trace:
I20260806 04:00:11.874080 32729 rpcz_store.cc:276] 0806 04:00:10.786211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.685441 (+899230us) service_pool.cc:227] Handling call
0806 04:00:11.708521 (+ 23080us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.708615 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.821968 (+113353us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.873153 (+ 51185us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.873623 (+ 470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.876008 32729 tablet_service.cc:1467] Processing CreateTablet for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 04:00:11.877359 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.872596 32465 rpcz_store.cc:276] 0806 04:00:10.795993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.769611 (+973618us) service_pool.cc:227] Handling call
0806 04:00:11.771150 (+ 1539us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.771238 (+ 88us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.838154 (+ 66916us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.871585 (+ 33431us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.872064 (+ 479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.880335 371 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:11.880852 371 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:11.854064 357 tablet_bootstrap.cc:492] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
W20260806 04:00:11.850255 32427 leader_election.cc:341] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.881202 371 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:11.886498 357 ts_tablet_manager.cc:1397] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.309s user 0.015s sys 0.003s
I20260806 04:00:11.881127 32465 tablet_service.cc:1467] Processing CreateTablet for tablet d24bcc94fb734c95849f4cb89c15ca8e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 04:00:11.875972 32460 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 49) took 1081 ms. Trace:
I20260806 04:00:11.896860 32449 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 56) took 1100 ms. Trace:
W20260806 04:00:11.897265 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:11.917877 32723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 55) took 1105 ms. Trace:
I20260806 04:00:11.920399 32723 rpcz_store.cc:276] 0806 04:00:10.812550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.822160 (+1009610us) service_pool.cc:227] Handling call
0806 04:00:11.861561 (+ 39401us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.861667 (+ 106us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.875532 (+ 13865us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.917341 (+ 41809us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.917813 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.903087 32725 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 50) took 1116 ms. Trace:
I20260806 04:00:11.921708 32451 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 52) took 1126 ms. Trace:
I20260806 04:00:11.921877 32725 rpcz_store.cc:276] 0806 04:00:10.787169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.733867 (+946698us) service_pool.cc:227] Handling call
0806 04:00:11.735572 (+ 1705us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.735660 (+ 88us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.812628 (+ 76968us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.902508 (+ 89880us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.903031 (+ 523us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5177}
I20260806 04:00:11.918051 357 raft_consensus.cc:348] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.923051 32723 tablet_service.cc:1467] Processing CreateTablet for tablet e870f9de1d6a4db4bbf149c6a785135c (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 04:00:11.923523 357 raft_consensus.cc:374] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.923849 357 raft_consensus.cc:729] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.923957 32725 tablet_service.cc:1467] Processing CreateTablet for tablet d24bcc94fb734c95849f4cb89c15ca8e (DEFAULT_TABLE table=test-workload [id=1530919f0c614d53a2009fb4cfbfa791]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 04:00:11.924479 32723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e870f9de1d6a4db4bbf149c6a785135c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.924588 357 consensus_queue.cc:260] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.899295 32453 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 55) took 1103 ms. Trace:
I20260806 04:00:11.925554 32725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24bcc94fb734c95849f4cb89c15ca8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.925760 32453 rpcz_store.cc:276] 0806 04:00:10.796357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.622128 (+825771us) service_pool.cc:227] Handling call
0806 04:00:11.624948 (+ 2820us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.630763 (+ 5815us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.891161 (+260398us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.898721 (+ 7560us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.899234 (+ 513us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.926764 357 ts_tablet_manager.cc:1428] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 04:00:11.903586 32729 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 56) took 1090 ms. Trace:
W20260806 04:00:11.905067 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 55325.9us
I20260806 04:00:11.906263 351 tablet_bootstrap.cc:492] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:11.928802 351 ts_tablet_manager.cc:1397] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.224s user 0.006s sys 0.008s
I20260806 04:00:11.929958 32729 rpcz_store.cc:276] 0806 04:00:10.812894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.875016 (+1062122us) service_pool.cc:227] Handling call
0806 04:00:11.876868 (+ 1852us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.876973 (+ 105us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.892670 (+ 15697us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.903101 (+ 10431us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.903530 (+ 429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.931515 351 raft_consensus.cc:348] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.932544 351 raft_consensus.cc:374] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.932857 351 raft_consensus.cc:729] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.933648 351 consensus_queue.cc:260] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.913368 32449 rpcz_store.cc:276] 0806 04:00:10.796648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.786944 (+990296us) service_pool.cc:227] Handling call
0806 04:00:11.788837 (+ 1893us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.788949 (+ 112us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.858504 (+ 69555us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.881422 (+ 22918us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.896782 (+ 15360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.907655 32721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 52) took 1119 ms. Trace:
I20260806 04:00:11.907891 32587 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41296 (request call id 56) took 1170 ms. Trace:
I20260806 04:00:11.942426 32587 rpcz_store.cc:276] 0806 04:00:10.737070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.607791 (+870721us) service_pool.cc:227] Handling call
0806 04:00:11.626734 (+ 18943us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.631404 (+ 4670us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.754673 (+123269us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.907383 (+152710us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.907832 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.943812 357 tablet_bootstrap.cc:492] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:11.910231 32457 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 50) took 1115 ms. Trace:
I20260806 04:00:11.945600 32457 rpcz_store.cc:276] 0806 04:00:10.794785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.654617 (+859832us) service_pool.cc:227] Handling call
0806 04:00:11.664176 (+ 9559us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.664385 (+ 209us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.833056 (+168671us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.909752 (+ 76696us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.910176 (+ 424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.914820 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24bcc94fb734c95849f4cb89c15ca8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:11.941766 32733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 44) took 1156 ms. Trace:
I20260806 04:00:11.912804 32460 rpcz_store.cc:276] 0806 04:00:10.794468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.636839 (+842371us) service_pool.cc:227] Handling call
0806 04:00:11.695591 (+ 58752us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.695681 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.856688 (+161007us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.875444 (+ 18756us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.875912 (+ 468us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.948253 32733 rpcz_store.cc:276] 0806 04:00:10.785283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.609454 (+824171us) service_pool.cc:227] Handling call
0806 04:00:11.661371 (+ 51917us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.661469 (+ 98us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.800770 (+139301us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.941258 (+140488us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.941711 (+ 453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.920034 352 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:11.950505 352 ts_tablet_manager.cc:1397] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.215s user 0.014s sys 0.006s
I20260806 04:00:11.950567 32454 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 57) took 1153 ms. Trace:
I20260806 04:00:11.950973 32454 rpcz_store.cc:276] 0806 04:00:10.796918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.789310 (+992392us) service_pool.cc:227] Handling call
0806 04:00:11.834701 (+ 45391us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.834812 (+ 111us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.908723 (+ 73911us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.950068 (+ 41345us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.950510 (+ 442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.901819 32452 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 53) took 1106 ms. Trace:
I20260806 04:00:11.922097 32451 rpcz_store.cc:276] 0806 04:00:10.795407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.704171 (+908764us) service_pool.cc:227] Handling call
0806 04:00:11.731128 (+ 26957us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.740491 (+ 9363us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.834040 (+ 93549us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.921294 (+ 87254us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.921660 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.904747 32736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 53) took 1116 ms. Trace:
I20260806 04:00:11.954684 32736 rpcz_store.cc:276] 0806 04:00:10.800980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.770174 (+969194us) service_pool.cc:227] Handling call
0806 04:00:11.771714 (+ 1540us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.771799 (+ 85us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.842287 (+ 70488us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.904248 (+ 61961us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.904690 (+ 442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.903997 32722 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 48) took 1117 ms. Trace:
I20260806 04:00:11.955760 32722 rpcz_store.cc:276] 0806 04:00:10.786531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.700142 (+913611us) service_pool.cc:227] Handling call
0806 04:00:11.701770 (+ 1628us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.701858 (+ 88us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.815819 (+113961us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.903530 (+ 87711us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.903939 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.954340 32452 rpcz_store.cc:276] 0806 04:00:10.795708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.739564 (+943856us) service_pool.cc:227] Handling call
0806 04:00:11.784279 (+ 44715us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.784368 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.890924 (+106556us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.901189 (+ 10265us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.901750 (+ 561us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.939572 351 ts_tablet_manager.cc:1428] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.010s user 0.003s sys 0.002s
I20260806 04:00:11.957993 351 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:11.958117 352 raft_consensus.cc:348] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.958885 352 raft_consensus.cc:374] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.959205 352 raft_consensus.cc:729] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.953437 32324 catalog_manager.cc:5495] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:11.959964 32725 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 58) took 1146 ms. Trace:
I20260806 04:00:11.958715 357 tablet_bootstrap.cc:654] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:11.960001 352 consensus_queue.cc:260] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.918036 32732 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 54) took 1106 ms. Trace:
I20260806 04:00:11.921670 32461 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 48) took 1135 ms. Trace:
I20260806 04:00:11.908990 32727 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 51) took 1121 ms. Trace:
I20260806 04:00:11.960654 32462 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 51) took 1165 ms. Trace:
I20260806 04:00:11.964375 32461 rpcz_store.cc:276] 0806 04:00:10.794063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.590353 (+796290us) service_pool.cc:227] Handling call
0806 04:00:11.740168 (+149815us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.741581 (+ 1413us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.849507 (+107926us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.921170 (+ 71663us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.921611 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.965482 32462 rpcz_store.cc:276] 0806 04:00:10.795099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.680624 (+885525us) service_pool.cc:227] Handling call
0806 04:00:11.731422 (+ 50798us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.731513 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.933438 (+201925us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.947760 (+ 14322us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.960592 (+ 12832us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.964732 32727 rpcz_store.cc:276] 0806 04:00:10.787463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.744972 (+957509us) service_pool.cc:227] Handling call
0806 04:00:11.789144 (+ 44172us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.789235 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.896437 (+107202us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.908648 (+ 12211us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.908946 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.960482 32725 rpcz_store.cc:276] 0806 04:00:10.813506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.923110 (+1109604us) service_pool.cc:227] Handling call
0806 04:00:11.924714 (+ 1604us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.924804 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.939526 (+ 14722us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.959358 (+ 19832us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.959908 (+ 550us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.969938 32734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 49) took 1183 ms. Trace:
I20260806 04:00:11.970623 352 ts_tablet_manager.cc:1428] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260806 04:00:11.970346 32734 rpcz_store.cc:276] 0806 04:00:10.786836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.718871 (+932035us) service_pool.cc:227] Handling call
0806 04:00:11.720572 (+ 1701us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.720767 (+ 195us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.833056 (+112289us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.969474 (+136418us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.969883 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.976845 352 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:11.976919 351 tablet_bootstrap.cc:654] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:11.964964 32732 rpcz_store.cc:276] 0806 04:00:10.812169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.780413 (+968244us) service_pool.cc:227] Handling call
0806 04:00:11.782042 (+ 1629us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.782132 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.891424 (+109292us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.917541 (+ 26117us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.917979 (+ 438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.964308 32721 rpcz_store.cc:276] 0806 04:00:10.787756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.764678 (+976922us) service_pool.cc:227] Handling call
0806 04:00:11.809029 (+ 44351us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.809126 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.895554 (+ 86428us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.907200 (+ 11646us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.907597 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.988267 357 tablet_bootstrap.cc:492] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:11.988824 357 ts_tablet_manager.cc:1397] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.006s
I20260806 04:00:11.989571 371 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.990000 32465 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59492 (request call id 58) took 1192 ms. Trace:
I20260806 04:00:11.990578 32465 rpcz_store.cc:276] 0806 04:00:10.797201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.880229 (+1083028us) service_pool.cc:227] Handling call
0806 04:00:11.914354 (+ 34125us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.914462 (+ 108us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.966187 (+ 51725us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:11.989517 (+ 23330us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:11.989939 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:11.992537 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:11.993110 32478 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:11.992707 357 raft_consensus.cc:348] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.994035 357 raft_consensus.cc:374] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:11.994163 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:11.994328 357 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:11.994392 352 tablet_bootstrap.cc:654] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
W20260806 04:00:11.995527 32682 leader_election.cc:341] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:11.996238 371 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:11.996500 357 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:11.999928 351 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.000264 32478 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:12.000551 351 ts_tablet_manager.cc:1397] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.010s
I20260806 04:00:12.001399 32683 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:12.002171 371 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:12.002629 371 raft_consensus.cc:686] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:12.003409 371 consensus_queue.cc:237] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.005069 32723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33964 (request call id 57) took 1191 ms. Trace:
I20260806 04:00:12.005465 32723 rpcz_store.cc:276] 0806 04:00:10.813199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:11.922051 (+1108852us) service_pool.cc:227] Handling call
0806 04:00:11.923919 (+ 1868us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 04:00:11.924038 (+ 119us) ts_tablet_manager.cc:676] Creating new metadata...
0806 04:00:11.991769 (+ 67731us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:12.004474 (+ 12705us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:12.005012 (+ 538us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:00:12.006435 352 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.006837 352 ts_tablet_manager.cc:1397] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.004s
I20260806 04:00:12.007339 357 ts_tablet_manager.cc:1428] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
I20260806 04:00:12.008320 357 tablet_bootstrap.cc:492] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:12.009223 352 raft_consensus.cc:348] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.009963 352 raft_consensus.cc:374] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.009501 351 raft_consensus.cc:348] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.010267 352 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.010404 351 raft_consensus.cc:374] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.010823 351 raft_consensus.cc:729] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.011129 352 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.011662 351 consensus_queue.cc:260] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.013872 352 ts_tablet_manager.cc:1428] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:00:12.014581 357 tablet_bootstrap.cc:654] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.014907 352 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.021680 352 tablet_bootstrap.cc:654] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.022055 351 ts_tablet_manager.cc:1428] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.021s user 0.005s sys 0.002s
I20260806 04:00:12.030680 352 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.031250 352 ts_tablet_manager.cc:1397] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260806 04:00:12.032896 32322 catalog_manager.cc:5495] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:12.034217 352 raft_consensus.cc:348] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.034813 357 tablet_bootstrap.cc:492] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:12.035086 352 raft_consensus.cc:374] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.035399 357 ts_tablet_manager.cc:1397] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.001s
I20260806 04:00:12.035493 352 raft_consensus.cc:729] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.036449 352 consensus_queue.cc:260] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.038142 357 raft_consensus.cc:348] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.038831 357 raft_consensus.cc:374] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.039093 352 ts_tablet_manager.cc:1428] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260806 04:00:12.039757 351 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:12.039108 357 raft_consensus.cc:729] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.039641 360 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.044850 360 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.045312 352 tablet_bootstrap.cc:492] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.045162 357 consensus_queue.cc:260] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.047192 360 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.048436 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.049348 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.049991 32612 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:12.050940 351 tablet_bootstrap.cc:654] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.051604 352 tablet_bootstrap.cc:654] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.053071 32427 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:00:12.054286 32428 leader_election.cc:341] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.056545 391 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.056842 352 tablet_bootstrap.cc:492] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.057307 352 ts_tablet_manager.cc:1397] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260806 04:00:12.057013 391 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.058769 360 raft_consensus.cc:2781] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.059015 391 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.059147 360 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.059605 360 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.059931 352 raft_consensus.cc:348] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.060688 352 raft_consensus.cc:374] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.060961 352 raft_consensus.cc:729] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.061326 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.061642 352 consensus_queue.cc:260] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:12.062757 32553 leader_election.cc:341] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.063943 352 ts_tablet_manager.cc:1428] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 04:00:12.064023 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.064946 352 tablet_bootstrap.cc:492] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.066921 360 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:12.067849 32553 leader_election.cc:341] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.069681 32553 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.070762 391 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.071192 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:12.071774 32612 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.071432 352 tablet_bootstrap.cc:654] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.068919 357 ts_tablet_manager.cc:1428] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.033s user 0.007s sys 0.004s
I20260806 04:00:12.076922 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:12.078540 32612 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
W20260806 04:00:12.078933 32428 leader_election.cc:341] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.079599 32427 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.079711 360 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.080189 357 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:12.080459 360 raft_consensus.cc:2781] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:12.080997 360 raft_consensus.cc:686] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:12.081918 360 consensus_queue.cc:237] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.084215 352 tablet_bootstrap.cc:492] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.084295 351 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.084766 352 ts_tablet_manager.cc:1397] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.002s
I20260806 04:00:12.084932 351 ts_tablet_manager.cc:1397] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.007s
I20260806 04:00:12.086732 357 tablet_bootstrap.cc:654] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.087148 352 raft_consensus.cc:348] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.087984 352 raft_consensus.cc:374] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.088287 352 raft_consensus.cc:729] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.088135 351 raft_consensus.cc:348] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.089074 351 raft_consensus.cc:374] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.089062 352 consensus_queue.cc:260] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.089578 351 raft_consensus.cc:729] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.090487 351 consensus_queue.cc:260] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.091770 352 ts_tablet_manager.cc:1428] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:00:12.092823 352 tablet_bootstrap.cc:492] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.098393 352 tablet_bootstrap.cc:654] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.098941 357 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:12.099428 357 ts_tablet_manager.cc:1397] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20260806 04:00:12.101308 351 ts_tablet_manager.cc:1428] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260806 04:00:12.102388 357 raft_consensus.cc:348] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.103185 357 raft_consensus.cc:374] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.103475 357 raft_consensus.cc:729] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.104339 357 consensus_queue.cc:260] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.109694 351 tablet_bootstrap.cc:492] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:12.115800 351 tablet_bootstrap.cc:654] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.116374 357 ts_tablet_manager.cc:1428] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.017s user 0.002s sys 0.004s
I20260806 04:00:12.117460 357 tablet_bootstrap.cc:492] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:12.115741 32322 catalog_manager.cc:5495] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:12.125244 352 tablet_bootstrap.cc:492] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.125759 352 ts_tablet_manager.cc:1397] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.016s
I20260806 04:00:12.125234 357 tablet_bootstrap.cc:654] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.128226 351 tablet_bootstrap.cc:492] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.128201 352 raft_consensus.cc:348] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.128875 351 ts_tablet_manager.cc:1397] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20260806 04:00:12.129001 352 raft_consensus.cc:374] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.129384 352 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.130029 352 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.131956 351 raft_consensus.cc:348] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.132491 391 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.132684 352 ts_tablet_manager.cc:1428] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:00:12.132817 351 raft_consensus.cc:374] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.133229 351 raft_consensus.cc:729] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.132982 391 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.134124 352 tablet_bootstrap.cc:492] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.133924 351 consensus_queue.cc:260] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.134899 391 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.137183 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.139832 352 tablet_bootstrap.cc:654] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.137185 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:00:12.141888 32553 leader_election.cc:341] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:12.142627 32553 leader_election.cc:341] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.142995 32553 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.148463 391 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.152952 371 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.153535 371 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.155603 406 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.156183 406 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.158126 406 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.159291 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.160043 32612 raft_consensus.cc:2379] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:12.160677 360 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.161114 360 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.161967 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.163131 360 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.163457 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.164289 32477 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:12.165628 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.166307 32683 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:00:12.167809 32428 leader_election.cc:341] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.168974 32682 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:12.170821 406 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.170725 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.171622 406 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.171206 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.155683 371 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.173008 357 tablet_bootstrap.cc:492] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
W20260806 04:00:12.172719 32427 leader_election.cc:341] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.172855 371 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.173652 357 ts_tablet_manager.cc:1397] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.008s
I20260806 04:00:12.173547 32427 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.172004 406 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.173956 351 ts_tablet_manager.cc:1428] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.045s user 0.021s sys 0.020s
I20260806 04:00:12.174657 360 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.176488 357 raft_consensus.cc:348] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.177199 357 raft_consensus.cc:374] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.177475 357 raft_consensus.cc:729] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.178140 357 consensus_queue.cc:260] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.180279 357 ts_tablet_manager.cc:1428] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:00:12.181794 406 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.182416 351 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
W20260806 04:00:12.183481 32682 leader_election.cc:341] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.183876 406 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.184947 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:12.185767 32477 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.172636 352 tablet_bootstrap.cc:492] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.185221 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:00:12.189392 32682 leader_election.cc:341] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.190032 351 tablet_bootstrap.cc:654] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.190133 352 ts_tablet_manager.cc:1397] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.056s user 0.016s sys 0.013s
I20260806 04:00:12.191634 32477 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:12.192703 32683 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:12.193405 371 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.194082 406 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:12.194727 440 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.194770 352 raft_consensus.cc:348] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.195626 352 raft_consensus.cc:374] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.195235 440 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.196132 352 raft_consensus.cc:729] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.196872 352 consensus_queue.cc:260] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.197265 357 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:12.197336 440 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.202230 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.202626 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.202986 32477 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:12.203969 32683 leader_election.cc:302] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:00:12.204921 32682 leader_election.cc:341] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.205606 352 ts_tablet_manager.cc:1428] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.014s user 0.002s sys 0.004s
I20260806 04:00:12.205637 440 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.206100 440 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.206424 440 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.206784 352 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.213353 406 raft_consensus.cc:686] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:12.215320 406 consensus_queue.cc:237] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.224964 440 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.226130 357 tablet_bootstrap.cc:654] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.227242 352 tablet_bootstrap.cc:654] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.230701 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:12.231412 32477 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.235667 351 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.236382 351 ts_tablet_manager.cc:1397] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.054s user 0.037s sys 0.014s
I20260806 04:00:12.237275 440 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.238656 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:00:12.240495 32682 leader_election.cc:341] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.244310 351 raft_consensus.cc:348] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.246971 32477 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:12.248112 351 raft_consensus.cc:374] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.248482 351 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.249181 351 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.249362 32683 leader_election.cc:302] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:12.251354 351 ts_tablet_manager.cc:1428] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 04:00:12.252012 406 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:12.252674 406 raft_consensus.cc:686] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:12.253759 406 consensus_queue.cc:237] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.247686 357 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:12.257135 351 tablet_bootstrap.cc:492] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:12.255709 32322 catalog_manager.cc:5495] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
W20260806 04:00:12.258070 32427 consensus_peers.cc:487] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Couldn't send request to peer dd7344d85a154539b7f8c484f2a76772. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:12.263080 357 ts_tablet_manager.cc:1397] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.010s
I20260806 04:00:12.265202 352 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.265661 352 ts_tablet_manager.cc:1397] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.031s
I20260806 04:00:12.266803 437 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.267347 437 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.269726 351 tablet_bootstrap.cc:654] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.270300 437 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.271826 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:12.273370 32553 leader_election.cc:341] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.275267 357 raft_consensus.cc:348] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.276122 357 raft_consensus.cc:374] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.276427 360 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.276815 352 raft_consensus.cc:348] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.276422 357 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.276908 360 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.277956 357 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.279302 351 tablet_bootstrap.cc:492] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.279822 351 ts_tablet_manager.cc:1397] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.007s
I20260806 04:00:12.281205 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.283154 351 raft_consensus.cc:348] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.283830 357 ts_tablet_manager.cc:1428] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260806 04:00:12.284171 351 raft_consensus.cc:374] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.284523 351 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.285122 357 tablet_bootstrap.cc:492] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:12.285401 351 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.284580 32322 catalog_manager.cc:5495] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:12.286586 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.287289 32612 raft_consensus.cc:2454] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:12.288134 351 ts_tablet_manager.cc:1428] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 04:00:12.289157 32427 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:12.289686 351 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:12.290653 380 raft_consensus.cc:2781] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.291222 380 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.291730 380 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.294840 352 raft_consensus.cc:374] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.295162 352 raft_consensus.cc:729] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.295900 352 consensus_queue.cc:260] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.297072 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.297766 357 tablet_bootstrap.cc:654] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.299374 351 tablet_bootstrap.cc:654] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
W20260806 04:00:12.299126 32428 leader_election.cc:341] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:12.300354 32553 leader_election.cc:341] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.301029 32553 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.301959 437 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.304489 406 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.305195 406 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.313485 406 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.315842 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:00:12.317706 32683 leader_election.cc:341] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.319252 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.319967 32612 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:12.321599 32682 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:12.323220 360 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.333298 406 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.333938 406 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.334343 406 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.335974 360 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:12.355536 380 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.362924 380 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.364097 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:12.365840 32612 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.375612 357 tablet_bootstrap.cc:492] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:12.376183 357 ts_tablet_manager.cc:1397] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.091s user 0.020s sys 0.015s
I20260806 04:00:12.380733 32612 raft_consensus.cc:2454] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:12.381841 357 raft_consensus.cc:348] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.382717 357 raft_consensus.cc:374] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.383244 357 raft_consensus.cc:729] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.382668 32427 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:12.384040 357 consensus_queue.cc:260] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.384063 352 ts_tablet_manager.cc:1428] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.118s user 0.026s sys 0.021s
I20260806 04:00:12.386502 352 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.383265 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:12.387562 454 raft_consensus.cc:2781] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:12.388720 454 raft_consensus.cc:686] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:12.390444 454 consensus_queue.cc:237] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:12.394279 32428 leader_election.cc:341] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.393107 351 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.393739 357 ts_tablet_manager.cc:1428] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.017s user 0.005s sys 0.003s
I20260806 04:00:12.395099 437 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.398042 351 ts_tablet_manager.cc:1397] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.109s user 0.027s sys 0.061s
I20260806 04:00:12.398257 380 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.397035 406 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.401515 357 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:12.400244 437 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.404364 351 raft_consensus.cc:348] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.404760 437 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.405149 351 raft_consensus.cc:374] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.405500 351 raft_consensus.cc:729] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.398897 380 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.413043 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.414162 351 consensus_queue.cc:260] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.415897 351 ts_tablet_manager.cc:1428] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.017s user 0.003s sys 0.009s
W20260806 04:00:12.418634 32553 leader_election.cc:341] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.419245 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:12.420639 32553 leader_election.cc:341] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.421046 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.422125 391 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.422318 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:00:12.424015 32683 leader_election.cc:341] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.424746 351 tablet_bootstrap.cc:492] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:12.426795 406 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.428582 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:12.429446 32612 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.432853 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.433634 32614 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:12.435212 351 tablet_bootstrap.cc:654] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.435637 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:12.436929 360 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.437356 360 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.437777 360 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.439056 352 tablet_bootstrap.cc:654] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
W20260806 04:00:12.450618 32682 consensus_peers.cc:487] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 -> Peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Couldn't send request to peer dd7344d85a154539b7f8c484f2a76772. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:12.457201 380 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.460523 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:12.464941 32428 leader_election.cc:341] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.470978 32324 catalog_manager.cc:5495] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:12.477595 360 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.479490 32612 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:12.480659 357 tablet_bootstrap.cc:654] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.480842 32682 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:12.482858 406 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:12.483454 406 raft_consensus.cc:686] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:12.481737 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:12.484493 32614 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.484786 406 consensus_queue.cc:237] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.487154 360 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.489655 351 tablet_bootstrap.cc:492] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.490167 351 ts_tablet_manager.cc:1397] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.020s
I20260806 04:00:12.490667 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:00:12.492425 32428 leader_election.cc:341] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.493739 351 raft_consensus.cc:348] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.494554 351 raft_consensus.cc:374] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.494835 351 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.495688 351 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.498193 351 ts_tablet_manager.cc:1428] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260806 04:00:12.498965 32614 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:12.499541 351 tablet_bootstrap.cc:492] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:12.500433 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.502082 360 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:12.502583 360 raft_consensus.cc:686] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:12.503634 360 consensus_queue.cc:237] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.517324 351 tablet_bootstrap.cc:654] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.528595 352 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.529079 352 ts_tablet_manager.cc:1397] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.143s user 0.004s sys 0.015s
I20260806 04:00:12.532243 352 raft_consensus.cc:348] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.532986 352 raft_consensus.cc:374] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.533259 352 raft_consensus.cc:729] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.534015 352 consensus_queue.cc:260] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.545501 380 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:12.551915 352 ts_tablet_manager.cc:1428] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.023s user 0.007s sys 0.004s
I20260806 04:00:12.552922 352 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.557152 357 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:12.557286 380 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.558076 380 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.557680 357 ts_tablet_manager.cc:1397] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.156s user 0.017s sys 0.007s
I20260806 04:00:12.562772 32326 catalog_manager.cc:5495] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:12.564043 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.563491 32324 catalog_manager.cc:5495] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:12.565955 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:12.565979 32428 leader_election.cc:341] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.566673 380 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:00:12.569778 32427 leader_election.cc:341] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.571249 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.572881 380 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.573645 351 tablet_bootstrap.cc:492] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.574120 352 tablet_bootstrap.cc:654] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.574100 351 ts_tablet_manager.cc:1397] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.025s
I20260806 04:00:12.581303 351 raft_consensus.cc:348] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.582041 351 raft_consensus.cc:374] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.582317 351 raft_consensus.cc:729] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.583312 351 consensus_queue.cc:260] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.586198 351 ts_tablet_manager.cc:1428] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260806 04:00:12.593169 351 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:12.592885 360 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:12.611081 357 raft_consensus.cc:348] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.611836 357 raft_consensus.cc:374] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.612151 357 raft_consensus.cc:729] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.612808 357 consensus_queue.cc:260] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.614830 357 ts_tablet_manager.cc:1428] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.054s user 0.004s sys 0.000s
I20260806 04:00:12.619203 351 tablet_bootstrap.cc:654] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.620160 406 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.620744 406 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.622781 357 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:12.626482 352 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:12.710529 352 ts_tablet_manager.cc:1397] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.158s user 0.030s sys 0.049s
I20260806 04:00:12.659122 440 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.625135 406 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.687428 360 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.714295 360 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.714996 352 raft_consensus.cc:348] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.715755 352 raft_consensus.cc:374] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.716027 352 raft_consensus.cc:729] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.716380 371 consensus_queue.cc:1035] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:12.716815 352 consensus_queue.cc:260] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.717492 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.718191 32610 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:12.719110 352 ts_tablet_manager.cc:1428] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260806 04:00:12.719852 32427 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:12.627457 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.720940 380 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.721292 380 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.721642 380 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.659566 32615 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:12.721553 32682 leader_election.cc:341] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.625862 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.712632 440 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.725281 32477 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:12.726856 32683 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:12.728164 371 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.728572 371 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.731256 351 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:12.731740 351 ts_tablet_manager.cc:1397] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.139s user 0.010s sys 0.011s
I20260806 04:00:12.731942 454 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.729496 371 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.732623 454 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.736017 352 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:12.736354 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.739032 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.738902 380 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.740020 32612 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:12.741001 440 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:12.756251 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.757284 32749 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:12.758795 32427 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:12.760777 450 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.761204 450 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.761566 450 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.799063 32478 raft_consensus.cc:2379] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:12.764246 454 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.737396 406 consensus_queue.cc:1035] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20260806 04:00:12.824340 454 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.853185 454 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.868927 351 raft_consensus.cc:348] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.869735 351 raft_consensus.cc:374] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:12.870031 351 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:12.870703 351 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.874871 391 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.876952 391 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.881333 357 tablet_bootstrap.cc:654] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:12.895013 406 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.895579 406 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.896901 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.868497 454 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.897713 406 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:12.900825 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.901557 32474 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:12.903151 32683 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:12.906929 406 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.897755 360 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.908116 406 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.908393 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.922152 380 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.925207 406 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.924008 440 consensus_queue.cc:1035] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260806 04:00:12.935544 437 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:00:12.937026 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.929015 391 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.929613 32612 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:12.939601 437 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.939790 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:12.944748 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:12.945710 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.947489 391 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.930380 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:12.948504 32614 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.933635 480 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:12.952478 480 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:12.954177 32683 consensus_peers.cc:487] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 -> Peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Couldn't send request to peer ae0b3e43e7a64f789e51018fd5518d23. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:12.955273 437 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.955430 480 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:12.959553 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.959491 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.964775 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:12.965385 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.965564 32749 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
W20260806 04:00:12.966126 32428 leader_election.cc:341] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.967469 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:12.969384 32746 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.972954 32682 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:12.972955 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.973860 32610 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
W20260806 04:00:12.973871 32428 consensus_peers.cc:487] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Couldn't send request to peer f7c202b562ce482db283463d7f02e136. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:12.977558 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:12.979090 480 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:12.920634 371 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:12.981602 32610 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.982034 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:12.985154 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:12.986804 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:12.987659 32474 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.989212 32612 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:12.989416 32682 consensus_peers.cc:487] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 -> Peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Couldn't send request to peer dd7344d85a154539b7f8c484f2a76772. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260806 04:00:12.989822 32428 consensus_peers.cc:487] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Couldn't send request to peer f7c202b562ce482db283463d7f02e136. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:12.989151 32427 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
W20260806 04:00:12.990880 32682 leader_election.cc:341] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:12.993144 479 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:12.993723 479 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:12.995251 479 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:12.988372 450 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.007694 32614 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:13.009872 371 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:13.015774 351 ts_tablet_manager.cc:1428] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.284s user 0.014s sys 0.017s
I20260806 04:00:13.016914 351 tablet_bootstrap.cc:492] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:13.031320 450 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:13.035300 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:13.035643 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:13.036480 32745 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:13.037227 357 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:13.037703 357 ts_tablet_manager.cc:1397] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.415s user 0.015s sys 0.026s
I20260806 04:00:13.042129 357 raft_consensus.cc:348] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.042833 357 raft_consensus.cc:374] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.043103 357 raft_consensus.cc:729] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.043731 357 consensus_queue.cc:260] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.037082 480 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:00:13.045766 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:13.053923 32614 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:13.056459 32745 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:13.056720 32427 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:13.058005 360 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:13.058044 32428 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:13.058717 360 raft_consensus.cc:686] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:13.060215 357 ts_tablet_manager.cc:1428] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20260806 04:00:13.060485 380 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:13.061076 357 tablet_bootstrap.cc:492] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:13.061160 380 raft_consensus.cc:686] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:13.046069 479 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.046941 406 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.067593 406 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:13.062451 380 consensus_queue.cc:237] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.059903 360 consensus_queue.cc:237] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.081449 352 tablet_bootstrap.cc:654] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.067215 32474 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:13.087107 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:13.087949 32615 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:13.088565 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:13.089391 32748 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:13.080538 32746 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:13.084683 479 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:13.097435 32683 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:13.098505 477 raft_consensus.cc:2741] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260806 04:00:13.113687 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:13.114487 32608 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:13.108966 406 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:13.108568 32748 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:13.116359 32428 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:13.115173 406 raft_consensus.cc:686] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:13.119405 406 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.107780 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:13.106144 32614 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:13.131888 32615 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:13.124647 479 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:13.142987 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:13.144176 479 raft_consensus.cc:686] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:13.145565 479 consensus_queue.cc:237] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.124677 32475 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:13.153712 357 tablet_bootstrap.cc:654] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.152928 32326 catalog_manager.cc:5495] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:13.154923 32326 catalog_manager.cc:5495] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:13.157351 406 consensus_queue.cc:1035] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:13.158592 32608 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:13.160835 450 consensus_queue.cc:1035] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 04:00:13.165082 32682 leader_election.cc:341] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:13.166275 351 tablet_bootstrap.cc:654] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.166828 406 consensus_queue.cc:1035] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:13.164276 32324 catalog_manager.cc:5495] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:13.169837 352 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:13.170336 352 ts_tablet_manager.cc:1397] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.435s user 0.030s sys 0.039s
I20260806 04:00:13.181011 352 raft_consensus.cc:348] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.181816 352 raft_consensus.cc:374] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.182093 352 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.190312 479 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:13.192282 352 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.197971 352 ts_tablet_manager.cc:1428] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.027s user 0.007s sys 0.002s
I20260806 04:00:13.199095 352 tablet_bootstrap.cc:492] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:13.217156 32475 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:13.219054 32683 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:13.221006 477 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:13.222018 477 raft_consensus.cc:686] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:13.223256 477 consensus_queue.cc:237] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:13.238087 500 log.cc:924] Time spent T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.052s user 0.000s sys 0.002s
I20260806 04:00:13.248095 352 tablet_bootstrap.cc:654] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.279790 357 tablet_bootstrap.cc:492] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:13.286922 357 ts_tablet_manager.cc:1397] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.226s user 0.023s sys 0.016s
I20260806 04:00:13.308558 351 tablet_bootstrap.cc:492] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:13.309046 351 ts_tablet_manager.cc:1397] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.292s user 0.021s sys 0.013s
I20260806 04:00:13.309656 357 raft_consensus.cc:348] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.310338 357 raft_consensus.cc:374] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.310588 357 raft_consensus.cc:729] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.321866 357 consensus_queue.cc:260] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.324291 357 ts_tablet_manager.cc:1428] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.037s user 0.010s sys 0.000s
I20260806 04:00:13.325194 357 tablet_bootstrap.cc:492] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:13.335173 351 raft_consensus.cc:348] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.335903 351 raft_consensus.cc:374] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.336195 351 raft_consensus.cc:729] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.336823 351 consensus_queue.cc:260] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.348017 32326 catalog_manager.cc:5495] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:13.350847 351 ts_tablet_manager.cc:1428] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.041s user 0.005s sys 0.002s
I20260806 04:00:13.351792 351 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:13.357002 352 tablet_bootstrap.cc:492] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:13.357478 352 ts_tablet_manager.cc:1397] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.159s user 0.023s sys 0.019s
I20260806 04:00:13.358877 357 tablet_bootstrap.cc:654] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.360366 351 tablet_bootstrap.cc:654] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.360526 352 raft_consensus.cc:348] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.361246 352 raft_consensus.cc:374] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.361508 352 raft_consensus.cc:729] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.362215 352 consensus_queue.cc:260] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.364814 352 ts_tablet_manager.cc:1428] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 04:00:13.365687 352 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:13.365144 32321 catalog_manager.cc:5495] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:13.376641 352 tablet_bootstrap.cc:654] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.378674 351 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:13.379207 351 ts_tablet_manager.cc:1397] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.008s
I20260806 04:00:13.382714 351 raft_consensus.cc:348] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.383474 351 raft_consensus.cc:374] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.383788 351 raft_consensus.cc:729] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.384583 351 consensus_queue.cc:260] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.385605 357 tablet_bootstrap.cc:492] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:13.386158 357 ts_tablet_manager.cc:1397] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.015s
I20260806 04:00:13.387255 351 ts_tablet_manager.cc:1428] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260806 04:00:13.388332 351 tablet_bootstrap.cc:492] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:13.389470 357 raft_consensus.cc:348] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.390307 357 raft_consensus.cc:374] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.390671 357 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.391491 357 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.395167 357 ts_tablet_manager.cc:1428] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 04:00:13.395538 352 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:13.399785 352 ts_tablet_manager.cc:1397] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.017s
I20260806 04:00:13.396179 357 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:13.398052 351 tablet_bootstrap.cc:654] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.403153 352 raft_consensus.cc:348] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.403640 352 raft_consensus.cc:374] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.403798 352 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.404290 352 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.405766 352 ts_tablet_manager.cc:1428] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:00:13.406427 32748 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:13.425056 352 tablet_bootstrap.cc:492] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:13.429868 357 tablet_bootstrap.cc:654] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.432657 352 tablet_bootstrap.cc:654] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.478499 357 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:13.479031 357 ts_tablet_manager.cc:1397] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.083s user 0.022s sys 0.025s
I20260806 04:00:13.482758 357 raft_consensus.cc:348] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.483717 357 raft_consensus.cc:374] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.484017 357 raft_consensus.cc:729] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.484946 357 consensus_queue.cc:260] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.487370 357 ts_tablet_manager.cc:1428] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.008s user 0.000s sys 0.008s
I20260806 04:00:13.488016 351 tablet_bootstrap.cc:492] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:13.488270 357 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:13.488644 351 ts_tablet_manager.cc:1397] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.101s user 0.015s sys 0.065s
I20260806 04:00:13.489886 352 tablet_bootstrap.cc:492] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:13.490327 352 ts_tablet_manager.cc:1397] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.066s user 0.015s sys 0.027s
I20260806 04:00:13.492295 351 raft_consensus.cc:348] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.492834 351 raft_consensus.cc:374] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.493037 351 raft_consensus.cc:729] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.494519 351 consensus_queue.cc:260] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.496966 351 ts_tablet_manager.cc:1428] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:00:13.496769 352 raft_consensus.cc:348] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.497584 352 raft_consensus.cc:374] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.497867 352 raft_consensus.cc:729] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.498397 351 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:13.498490 352 consensus_queue.cc:260] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.499964 357 tablet_bootstrap.cc:654] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.500947 352 ts_tablet_manager.cc:1428] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260806 04:00:13.502110 352 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:13.510769 352 tablet_bootstrap.cc:654] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.514760 351 tablet_bootstrap.cc:654] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.520593 380 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:13.528982 357 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:13.528895 454 consensus_queue.cc:1035] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:13.529510 357 ts_tablet_manager.cc:1397] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.041s user 0.020s sys 0.000s
I20260806 04:00:13.533321 357 raft_consensus.cc:348] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.534040 357 raft_consensus.cc:374] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.534284 357 raft_consensus.cc:729] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.534977 357 consensus_queue.cc:260] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.534744 489 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.186s
I20260806 04:00:13.722661 351 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:13.723152 351 ts_tablet_manager.cc:1397] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.225s user 0.038s sys 0.040s
I20260806 04:00:13.658741 479 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:13.724690 479 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.727535 351 raft_consensus.cc:348] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.722584 357 ts_tablet_manager.cc:1428] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.193s user 0.010s sys 0.000s
I20260806 04:00:13.728471 351 raft_consensus.cc:374] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.728734 351 raft_consensus.cc:729] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.729100 357 tablet_bootstrap.cc:492] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:13.729277 351 consensus_queue.cc:260] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.626071 454 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:13.564111 477 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:13.731583 477 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.732520 352 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:13.732965 352 ts_tablet_manager.cc:1397] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.231s user 0.017s sys 0.019s
I20260806 04:00:13.730713 539 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:13.734624 477 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:13.734221 539 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.735891 352 raft_consensus.cc:348] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.736570 352 raft_consensus.cc:374] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:13.736634 351 ts_tablet_manager.cc:1428] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260806 04:00:13.736873 352 raft_consensus.cc:729] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:13.737197 539 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:13.737505 352 consensus_queue.cc:260] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.738081 479 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:13.739166 352 ts_tablet_manager.cc:1428] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:00:13.737505 351 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:13.660075 380 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:13.740200 380 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.730378 454 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.742383 352 tablet_bootstrap.cc:492] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:13.742934 492 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:13.744267 406 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:13.747807 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:13.748278 32609 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.059911 454 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:00:13.798422 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 92881.1us
I20260806 04:00:14.060896 492 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.061386 539 consensus_queue.cc:1035] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:14.062721 539 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:14.063644 406 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:13.920733 380 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.057284 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:00:13.779690 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 52882.5us
I20260806 04:00:14.068713 351 tablet_bootstrap.cc:654] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:13.796614 450 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.298s
W20260806 04:00:14.082747 32682 leader_election.cc:341] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:13.825476 541 consensus_queue.cc:1035] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.285s
I20260806 04:00:14.082928 547 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:14.081476 544 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:14.084533 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.085184 544 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.084522 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:14.084872 547 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.084951 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.087682 492 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.092163 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:14.084967 550 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:14.087765 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:14.095676 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:14.095510 550 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:14.096180 32553 leader_election.cc:341] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:14.096311 32477 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:14.096554 547 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.097625 32553 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:14.088898 406 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:14.092507 492 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:14.101692 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:14.102697 32614 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:14.100605 32475 raft_consensus.cc:2379] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 1: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:00:14.099514 544 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.100806 547 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:14.106092 352 tablet_bootstrap.cc:654] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:14.102056 492 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.106665 547 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:14.107933 547 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.108758 357 tablet_bootstrap.cc:654] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:14.111456 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.113355 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:14.113502 492 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.113921 32475 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:14.114961 32553 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:14.115902 550 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:14.115968 492 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:14.116530 492 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:14.120289 492 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.121176 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:14.125073 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 296726us
I20260806 04:00:14.125192 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:14.147773 32746 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:14.148133 32428 leader_election.cc:341] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:14.148988 32428 leader_election.cc:341] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:14.150776 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:14.155997 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.156903 542 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 04:00:14.166450 32428 leader_election.cc:341] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:14.172499 352 tablet_bootstrap.cc:492] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:14.173202 352 ts_tablet_manager.cc:1397] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.431s user 0.038s sys 0.073s
I20260806 04:00:14.178323 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:14.179219 32474 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:14.180943 357 tablet_bootstrap.cc:492] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:14.181632 357 ts_tablet_manager.cc:1397] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.453s user 0.009s sys 0.037s
I20260806 04:00:14.171660 547 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.184672 32614 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:14.208488 357 raft_consensus.cc:348] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.209028 357 raft_consensus.cc:374] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:14.209244 357 raft_consensus.cc:729] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:14.209774 357 consensus_queue.cc:260] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.190481 492 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.202230 542 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:14.214609 547 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.225546 32605 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.228747 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:14.230113 357 ts_tablet_manager.cc:1428] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.048s user 0.004s sys 0.003s
I20260806 04:00:14.233107 351 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:14.234666 351 ts_tablet_manager.cc:1397] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.497s user 0.015s sys 0.003s
I20260806 04:00:14.235580 357 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:14.217414 542 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.263057 32604 raft_consensus.cc:2379] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:14.260617 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:14.260851 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:14.279467 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:14.571560 351 raft_consensus.cc:348] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.572889 351 raft_consensus.cc:374] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:14.573207 351 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:14.573953 351 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:14.271878 32427 leader_election.cc:341] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:14.541736 569 log.cc:924] Time spent T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.178s user 0.000s sys 0.002s
I20260806 04:00:14.411688 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.577440 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:14.349522 492 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.579960 351 ts_tablet_manager.cc:1428] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.345s user 0.011s sys 0.046s
I20260806 04:00:14.511929 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:14.584440 351 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:14.574246 32478 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.399251 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.481488 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:14.419425 548 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:00:14.492583 32683 leader_election.cc:341] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:14.282140 542 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:00:14.319577 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 46382.2us
I20260806 04:00:14.591446 352 raft_consensus.cc:348] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.592285 352 raft_consensus.cc:374] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:14.297343 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.298985 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.593677 352 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:14.597075 352 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.594873 32475 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.594332 539 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.416780 545 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.264s
I20260806 04:00:14.592922 32745 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:14.609997 352 ts_tablet_manager.cc:1428] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.436s user 0.023s sys 0.018s
I20260806 04:00:14.611744 352 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:14.594020 32683 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:14.628911 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.636530 550 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.646842 545 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:00:14.651974 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 385279us
I20260806 04:00:14.601184 548 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:14.657147 571 log.cc:924] Time spent T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260806 04:00:14.653782 545 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:14.653010 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 163008us
W20260806 04:00:14.659610 575 log.cc:924] Time spent T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20260806 04:00:14.755569 32478 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:14.758044 351 tablet_bootstrap.cc:654] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:14.758638 32553 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:14.761487 492 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:14.761965 492 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:14.762815 492 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.764657 32475 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:14.766073 32553 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:14.767966 547 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:14.768231 545 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:14.759050 583 consensus_queue.cc:1035] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260806 04:00:14.750993 32427 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
W20260806 04:00:14.754704 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 47419us
W20260806 04:00:14.706820 32428 leader_election.cc:341] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:14.771154 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 64383.7us
I20260806 04:00:14.771962 580 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:14.772409 580 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:14.772159 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:14.772727 580 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.775527 32683 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:14.777390 32427 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:14.778167 542 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.779896 548 leader_election.cc:288] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.780385 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:14.780879 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:14.781294 32615 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:14.783892 477 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:14.783198 352 tablet_bootstrap.cc:654] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:14.784322 477 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:14.784974 477 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.793778 547 raft_consensus.cc:686] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:14.795082 547 consensus_queue.cc:237] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.805985 32683 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:14.807865 357 tablet_bootstrap.cc:654] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:14.782033 32427 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:00:14.810889 32553 leader_election.cc:341] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:14.811543 32553 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:14.808156 583 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.818539 32682 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:14.819242 380 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.823133 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:14.825618 492 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.835153 583 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:14.845805 32475 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:14.841377 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:14.844484 583 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:14.853559 583 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:14.844262 32553 leader_election.cc:341] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:14.857273 351 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:14.857776 351 ts_tablet_manager.cc:1397] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.274s user 0.016s sys 0.022s
I20260806 04:00:14.842133 580 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:14.862953 32553 leader_election.cc:341] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:14.869333 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 57347.2us
I20260806 04:00:14.880133 32321 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:14.885582 580 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.887048 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:14.888528 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:14.889635 590 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.890291 32553 leader_election.cc:302] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:14.891443 590 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:00:14.875150 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:14.904505 352 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:14.905193 352 ts_tablet_manager.cc:1397] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.294s user 0.013s sys 0.016s
I20260806 04:00:14.905321 357 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:14.896570 32474 raft_consensus.cc:2379] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:14.906720 357 ts_tablet_manager.cc:1397] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.671s user 0.014s sys 0.039s
I20260806 04:00:14.897060 477 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.916731 32749 raft_consensus.cc:2379] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 1: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:00:14.919507 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:14.926110 351 raft_consensus.cc:348] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.927026 351 raft_consensus.cc:374] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:14.927372 351 raft_consensus.cc:729] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:14.928226 351 consensus_queue.cc:260] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.935523 351 ts_tablet_manager.cc:1428] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.077s user 0.008s sys 0.000s
W20260806 04:00:14.914572 32427 leader_election.cc:341] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:14.925566 547 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:14.936947 351 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:14.923398 477 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:14.941672 590 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:14.956535 590 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.942919 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:14.958202 32474 raft_consensus.cc:2379] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:14.959782 590 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.947901 583 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.960008 32683 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:14.951611 547 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:14.961261 32746 raft_consensus.cc:2379] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 1: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:00:14.975342 583 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:14.964358 477 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.975728 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:14.976962 32474 raft_consensus.cc:2379] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:14.978891 547 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:14.963989 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:14.979638 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:14.980569 32615 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:14.984434 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 72052.2us
I20260806 04:00:14.989890 539 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:14.993306 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:15.000231 380 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.001041 380 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.006366 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.007294 32746 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:15.009430 32428 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:15.010834 380 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.011642 450 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:15.012252 450 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:15.012826 450 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:15.019711 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 73058.8us
I20260806 04:00:15.026211 357 raft_consensus.cc:348] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.026736 357 raft_consensus.cc:374] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.026970 357 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.023075 32615 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:15.086249 351 tablet_bootstrap.cc:654] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.070530 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.064419 352 raft_consensus.cc:348] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.099663 352 raft_consensus.cc:374] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.100064 352 raft_consensus.cc:729] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.099884 32682 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
W20260806 04:00:15.100412 32553 leader_election.cc:341] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.077775 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.064165 357 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.101439 583 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:15.101145 352 consensus_queue.cc:260] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.101892 583 raft_consensus.cc:686] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:15.103010 357 ts_tablet_manager.cc:1428] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.196s user 0.009s sys 0.002s
I20260806 04:00:15.102778 583 consensus_queue.cc:237] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.103581 352 ts_tablet_manager.cc:1428] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.198s user 0.021s sys 0.071s
I20260806 04:00:15.103756 357 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:15.104453 352 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:15.036108 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.105584 32478 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:15.064967 32427 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.030189 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.098361 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:15.101871 32553 leader_election.cc:341] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.111783 357 tablet_bootstrap.cc:654] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.112044 32553 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.110739 352 tablet_bootstrap.cc:654] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.110744 380 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260806 04:00:15.111452 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 46617.2us
I20260806 04:00:15.113549 450 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.116113 547 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.117377 450 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.119087 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:00:15.119254 32427 leader_election.cc:341] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.119951 32746 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.119107 32553 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:15.122427 547 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 04:00:15.122659 32553 leader_election.cc:341] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.122927 547 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:15.123346 547 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.134577 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:00:15.137046 32427 leader_election.cc:341] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.137269 32746 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:15.141948 32428 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:15.142005 547 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.144392 380 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:15.145176 380 raft_consensus.cc:686] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:15.146574 380 consensus_queue.cc:237] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.148763 547 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.150830 352 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:15.151309 352 ts_tablet_manager.cc:1397] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.013s
I20260806 04:00:15.151181 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:15.151791 32475 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.154614 352 raft_consensus.cc:348] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.155292 352 raft_consensus.cc:374] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.155560 352 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.156486 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:15.156360 352 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.157437 351 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
W20260806 04:00:15.158263 32553 leader_election.cc:341] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.158641 352 ts_tablet_manager.cc:1428] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260806 04:00:15.159042 351 ts_tablet_manager.cc:1397] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.222s user 0.018s sys 0.022s
I20260806 04:00:15.159523 352 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:15.161659 32475 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:15.161911 351 raft_consensus.cc:348] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.163285 32553 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.164821 547 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:15.165477 547 raft_consensus.cc:686] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:15.166685 547 consensus_queue.cc:237] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.169713 352 tablet_bootstrap.cc:654] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.177217 357 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:15.177716 357 ts_tablet_manager.cc:1397] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.002s
I20260806 04:00:15.162592 351 raft_consensus.cc:374] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.180122 351 raft_consensus.cc:729] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.181144 351 consensus_queue.cc:260] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.182315 357 raft_consensus.cc:348] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.183486 357 raft_consensus.cc:374] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.183792 357 raft_consensus.cc:729] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.184458 357 consensus_queue.cc:260] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.203687 357 ts_tablet_manager.cc:1428] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260806 04:00:15.203763 352 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:15.205717 352 ts_tablet_manager.cc:1397] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.015s
I20260806 04:00:15.205060 351 ts_tablet_manager.cc:1428] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.046s user 0.002s sys 0.004s
I20260806 04:00:15.206826 351 tablet_bootstrap.cc:492] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:15.207567 357 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
W20260806 04:00:15.218402 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:15.219010 351 tablet_bootstrap.cc:654] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.224548 352 raft_consensus.cc:348] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.234771 352 raft_consensus.cc:374] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.235122 352 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.236645 352 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.241202 352 ts_tablet_manager.cc:1428] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.035s user 0.011s sys 0.006s
I20260806 04:00:15.242133 352 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:15.242177 357 tablet_bootstrap.cc:654] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.257395 590 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.280151 352 tablet_bootstrap.cc:654] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.288553 380 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.283870 32324 catalog_manager.cc:5495] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:15.298741 351 tablet_bootstrap.cc:492] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:15.299606 351 ts_tablet_manager.cc:1397] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.011s
I20260806 04:00:15.292317 380 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.304565 380 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.304814 547 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.309834 351 raft_consensus.cc:348] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.310544 351 raft_consensus.cc:374] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.310756 351 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.311360 351 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.311825 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.313341 351 ts_tablet_manager.cc:1428] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20260806 04:00:15.313177 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:15.313501 32428 leader_election.cc:341] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.314237 351 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
W20260806 04:00:15.316424 32427 leader_election.cc:341] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.332341 583 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.317101 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.347115 583 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.348070 380 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.351194 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.351998 32475 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:15.353088 583 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:15.353677 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:15.317250 357 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:15.355294 357 ts_tablet_manager.cc:1397] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.149s user 0.014s sys 0.051s
I20260806 04:00:15.355247 32682 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:15.356676 32746 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.356860 351 tablet_bootstrap.cc:654] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.359114 357 raft_consensus.cc:348] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.360385 357 raft_consensus.cc:374] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.360651 357 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.363451 583 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.363545 492 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:15.362836 357 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.368532 351 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:15.369182 351 ts_tablet_manager.cc:1397] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.026s
I20260806 04:00:15.370688 357 ts_tablet_manager.cc:1428] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
I20260806 04:00:15.371605 357 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:15.371937 351 raft_consensus.cc:348] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.372699 351 raft_consensus.cc:374] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.372988 351 raft_consensus.cc:729] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.390033 351 consensus_queue.cc:260] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.398737 351 ts_tablet_manager.cc:1428] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.029s user 0.004s sys 0.001s
I20260806 04:00:15.396524 492 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.400413 492 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.400497 547 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.404532 352 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:15.404991 352 ts_tablet_manager.cc:1397] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.163s user 0.018s sys 0.032s
I20260806 04:00:15.406286 492 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.409721 380 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.410518 351 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:15.410295 380 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.410866 352 raft_consensus.cc:348] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.411525 352 raft_consensus.cc:374] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.411787 352 raft_consensus.cc:729] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.412390 352 consensus_queue.cc:260] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.414887 380 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.416819 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.417094 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:15.417737 32609 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:15.417608 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.418851 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.419085 32746 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
W20260806 04:00:15.420523 32553 leader_election.cc:341] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.421073 352 ts_tablet_manager.cc:1428] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20260806 04:00:15.420691 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:00:15.421474 32553 leader_election.cc:341] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.422051 352 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:15.422091 32553 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.422646 450 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:15.423002 450 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:15.423362 450 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.427345 492 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.425825 547 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.430338 547 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.430966 351 tablet_bootstrap.cc:654] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.432878 547 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.434904 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.436297 352 tablet_bootstrap.cc:654] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.438215 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.439391 32746 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:15.441632 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.445529 635 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.438588 32321 catalog_manager.cc:5495] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:15.451679 357 tablet_bootstrap.cc:654] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.451798 450 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.455277 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:15.455585 450 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.456902 32615 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.464299 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:15.465130 32746 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.470183 32615 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:15.472095 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:15.474176 380 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:15.477123 380 raft_consensus.cc:686] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:15.478940 380 consensus_queue.cc:237] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.478995 32326 catalog_manager.cc:5495] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:15.481297 32326 catalog_manager.cc:5495] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:15.489990 352 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:15.492304 32746 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:15.493582 635 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.499118 351 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:15.499614 352 ts_tablet_manager.cc:1397] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.014s
I20260806 04:00:15.500602 351 ts_tablet_manager.cc:1397] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.090s user 0.016s sys 0.016s
I20260806 04:00:15.507421 352 raft_consensus.cc:348] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.508154 352 raft_consensus.cc:374] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.508421 352 raft_consensus.cc:729] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.508972 352 consensus_queue.cc:260] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.511565 352 ts_tablet_manager.cc:1428] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260806 04:00:15.513610 351 raft_consensus.cc:348] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.514377 351 raft_consensus.cc:374] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.514643 351 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.515424 351 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.523413 352 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:15.530786 32326 catalog_manager.cc:5495] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:15.539482 357 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:15.540755 450 consensus_queue.cc:1035] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:15.540897 351 ts_tablet_manager.cc:1428] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20260806 04:00:15.541805 357 ts_tablet_manager.cc:1397] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.170s user 0.019s sys 0.019s
I20260806 04:00:15.545001 351 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:15.553368 635 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.553884 635 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.558014 635 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.559667 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.559706 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.562268 32475 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:15.563227 357 raft_consensus.cc:348] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.564484 357 raft_consensus.cc:374] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 04:00:15.564291 32553 leader_election.cc:341] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.564934 357 raft_consensus.cc:729] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.565493 32553 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.565724 357 consensus_queue.cc:260] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.566850 635 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:15.567339 635 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:15.567793 635 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.568388 357 ts_tablet_manager.cc:1428] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.026s user 0.006s sys 0.002s
I20260806 04:00:15.568634 450 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.569460 357 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:15.569808 351 tablet_bootstrap.cc:654] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.569309 450 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.570521 352 tablet_bootstrap.cc:654] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.572536 583 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.574512 583 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.576319 545 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.576833 545 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.577765 583 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:15.578939 450 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.581012 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.580586 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:15.581871 32749 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:15.582098 32605 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:15.584482 32682 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:15.584777 32428 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:15.585601 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:15.586110 450 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:15.586249 32609 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:15.586668 450 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:15.587167 450 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.588277 545 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:15.591368 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.591150 635 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.592247 32474 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:15.593616 545 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:15.594137 545 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:15.594233 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:15.594619 545 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.596307 32682 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:15.597726 583 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.598366 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.601198 590 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.612790 450 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.614661 32475 raft_consensus.cc:2379] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:15.618654 454 consensus_queue.cc:1035] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.620532 477 consensus_queue.cc:1035] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.624294 635 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.625953 357 tablet_bootstrap.cc:654] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.630715 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:15.632210 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:00:15.636705 32553 leader_election.cc:341] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.636795 32475 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.632207 583 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.641973 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:15.642549 352 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:15.642621 32615 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.643078 352 ts_tablet_manager.cc:1397] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.120s user 0.012s sys 0.007s
I20260806 04:00:15.656241 352 raft_consensus.cc:348] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.657599 352 raft_consensus.cc:374] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.657913 352 raft_consensus.cc:729] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.658699 352 consensus_queue.cc:260] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.662900 450 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.654053 545 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.664215 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:15.665509 32749 raft_consensus.cc:2379] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 1: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:00:15.667501 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:15.668412 32478 raft_consensus.cc:2379] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:15.668594 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:15.669296 635 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.670287 32682 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:15.671448 583 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.673053 352 ts_tablet_manager.cc:1428] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.030s user 0.005s sys 0.004s
I20260806 04:00:15.674010 352 tablet_bootstrap.cc:492] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:15.666599 545 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:15.681216 635 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.683706 32475 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:15.691414 450 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.694622 450 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.704377 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:15.691769 635 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:15.710628 32427 leader_election.cc:341] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.704667 32553 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.713471 590 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:15.713505 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.714143 590 raft_consensus.cc:686] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:15.709728 450 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.716300 352 tablet_bootstrap.cc:654] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.718170 351 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:15.718518 351 ts_tablet_manager.cc:1397] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.174s user 0.020s sys 0.027s
W20260806 04:00:15.720978 32428 leader_election.cc:341] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.721599 32428 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.723591 450 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.724447 357 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:15.724903 357 ts_tablet_manager.cc:1397] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.156s user 0.005s sys 0.016s
I20260806 04:00:15.726028 351 raft_consensus.cc:348] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.726722 351 raft_consensus.cc:374] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.726965 351 raft_consensus.cc:729] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.727142 357 raft_consensus.cc:348] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.728444 357 raft_consensus.cc:374] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.727571 351 consensus_queue.cc:260] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.728742 357 raft_consensus.cc:729] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.728502 583 consensus_queue.cc:1035] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:15.729765 351 ts_tablet_manager.cc:1428] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260806 04:00:15.729442 357 consensus_queue.cc:260] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.730625 351 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:15.731671 357 ts_tablet_manager.cc:1428] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260806 04:00:15.732503 357 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:15.735690 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:15.717612 635 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.715443 590 consensus_queue.cc:237] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.736469 32475 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:15.747121 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:15.749380 32553 leader_election.cc:341] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.757018 32553 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.758190 352 tablet_bootstrap.cc:492] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:15.758682 352 ts_tablet_manager.cc:1397] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.085s user 0.013s sys 0.007s
I20260806 04:00:15.781662 357 tablet_bootstrap.cc:654] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.783408 351 tablet_bootstrap.cc:654] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.776408 492 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:15.792680 32615 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:15.787185 492 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:15.794724 492 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:15.794945 32427 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.797539 380 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:15.798113 380 raft_consensus.cc:686] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:15.804409 352 raft_consensus.cc:348] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.805109 352 raft_consensus.cc:374] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.805365 352 raft_consensus.cc:729] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.805899 380 consensus_queue.cc:237] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.808065 352 consensus_queue.cc:260] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.813966 32326 catalog_manager.cc:5495] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:15.824265 352 ts_tablet_manager.cc:1428] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.065s user 0.011s sys 0.000s
I20260806 04:00:15.825281 352 tablet_bootstrap.cc:492] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:15.840212 32326 catalog_manager.cc:5495] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:15.841722 357 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:15.842121 357 ts_tablet_manager.cc:1397] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.110s user 0.015s sys 0.013s
I20260806 04:00:15.847074 539 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.847630 539 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.849005 357 raft_consensus.cc:348] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.850646 357 raft_consensus.cc:374] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.850903 357 raft_consensus.cc:729] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.851536 351 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:15.851483 357 consensus_queue.cc:260] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.852058 351 ts_tablet_manager.cc:1397] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.015s
I20260806 04:00:15.852900 352 tablet_bootstrap.cc:654] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.853257 357 ts_tablet_manager.cc:1428] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260806 04:00:15.854697 357 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:15.857307 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.859133 492 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.861115 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:15.862231 32428 leader_election.cc:341] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.862709 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:15.863358 32475 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:15.863873 32427 leader_election.cc:341] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.864476 32427 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.857263 539 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.865434 380 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:15.862257 351 raft_consensus.cc:348] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.866233 351 raft_consensus.cc:374] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:15.866510 351 raft_consensus.cc:729] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:15.868551 351 consensus_queue.cc:260] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.875438 380 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.895118 351 ts_tablet_manager.cc:1428] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.043s user 0.002s sys 0.004s
I20260806 04:00:15.896020 351 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:15.901127 357 tablet_bootstrap.cc:654] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:15.901537 380 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:15.913095 477 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.913898 477 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.915786 492 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:15.915068 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:00:15.924022 32553 leader_election.cc:341] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:15.915411 32475 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:15.937541 477 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:15.951725 583 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.936297 492 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:15.991844 492 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.975229 583 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:15.998749 583 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:15.981654 351 tablet_bootstrap.cc:654] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.003222 492 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:00:15.973455 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 51013.3us
I20260806 04:00:16.006371 351 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:16.006832 351 ts_tablet_manager.cc:1397] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.111s user 0.012s sys 0.000s
I20260806 04:00:16.009102 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.009961 32553 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:15.986020 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.013420 590 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:16.006016 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.011090 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:15.924265 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.014138 590 raft_consensus.cc:686] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:16.013355 32615 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:16.024025 32682 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:16.014827 351 raft_consensus.cc:348] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.015053 32474 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:16.015390 590 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.018806 32475 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:16.008952 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:15.935570 539 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:00:16.021548 32553 leader_election.cc:341] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.014623 352 tablet_bootstrap.cc:492] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:16.025751 351 raft_consensus.cc:374] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.025238 477 raft_consensus.cc:2781] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.031118 352 ts_tablet_manager.cc:1397] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.206s user 0.014s sys 0.014s
I20260806 04:00:16.033106 32683 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:16.037343 352 raft_consensus.cc:348] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.038081 352 raft_consensus.cc:374] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.038329 352 raft_consensus.cc:729] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.039005 352 consensus_queue.cc:260] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.039994 477 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.040566 477 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.042276 357 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:16.042778 583 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.042800 357 ts_tablet_manager.cc:1397] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.188s user 0.011s sys 0.005s
I20260806 04:00:16.031112 351 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.051066 352 ts_tablet_manager.cc:1428] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
W20260806 04:00:16.051213 32553 leader_election.cc:341] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.052307 352 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:16.052168 32553 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:16.053884 351 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.057246 351 ts_tablet_manager.cc:1428] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.050s user 0.003s sys 0.005s
I20260806 04:00:16.056152 660 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:16.049856 539 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.059250 351 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:16.048767 477 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.071061 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.072072 32615 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:16.073925 32427 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:16.075080 380 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.075619 380 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.076074 380 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.080698 539 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.088590 357 raft_consensus.cc:348] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.090056 380 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.091578 32326 catalog_manager.cc:5495] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:16.097575 357 raft_consensus.cc:374] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.098270 357 raft_consensus.cc:729] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.099201 357 consensus_queue.cc:260] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.094587 352 tablet_bootstrap.cc:654] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.095957 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.100345 32746 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:16.097000 583 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:16.103991 357 ts_tablet_manager.cc:1428] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.055s user 0.002s sys 0.006s
I20260806 04:00:16.105170 357 tablet_bootstrap.cc:492] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:16.107385 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:16.108201 32615 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.109193 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:16.109961 32475 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.109896 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:16.110713 32608 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.119882 477 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.121461 32608 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:16.123132 32682 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:16.125067 477 raft_consensus.cc:2781] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:16.125609 477 raft_consensus.cc:686] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:16.126633 32475 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:16.126727 477 consensus_queue.cc:237] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.131625 380 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.137022 32615 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:16.138441 32427 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:16.138784 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:16.139554 380 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:16.139554 32746 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.142701 351 tablet_bootstrap.cc:654] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.142349 32326 catalog_manager.cc:5495] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:16.143857 380 raft_consensus.cc:686] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:16.145340 380 consensus_queue.cc:237] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.146771 357 tablet_bootstrap.cc:654] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.151300 352 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:16.151746 352 ts_tablet_manager.cc:1397] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.007s
I20260806 04:00:16.152123 351 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:16.152606 351 ts_tablet_manager.cc:1397] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.014s
I20260806 04:00:16.155820 351 raft_consensus.cc:348] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.156908 351 raft_consensus.cc:374] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.157209 351 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.157999 351 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.163902 351 ts_tablet_manager.cc:1428] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260806 04:00:16.165555 351 tablet_bootstrap.cc:492] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:16.167191 32746 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:16.162354 352 raft_consensus.cc:348] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.167932 352 raft_consensus.cc:374] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.168234 352 raft_consensus.cc:729] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.169925 352 consensus_queue.cc:260] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.177026 351 tablet_bootstrap.cc:654] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.177676 352 ts_tablet_manager.cc:1428] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.026s user 0.009s sys 0.001s
I20260806 04:00:16.178619 352 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:16.180223 357 tablet_bootstrap.cc:492] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:16.180716 357 ts_tablet_manager.cc:1397] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.076s user 0.017s sys 0.002s
I20260806 04:00:16.181874 450 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.182389 450 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.184201 357 raft_consensus.cc:348] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.185052 357 raft_consensus.cc:374] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.185341 357 raft_consensus.cc:729] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.186208 357 consensus_queue.cc:260] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.188247 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.189278 357 ts_tablet_manager.cc:1428] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 04:00:16.190596 357 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:16.191242 352 tablet_bootstrap.cc:654] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.185385 450 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:00:16.191756 32427 leader_election.cc:341] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.195568 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:16.197268 32428 leader_election.cc:341] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.197793 32428 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:16.198416 32321 catalog_manager.cc:5495] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:16.199061 450 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:16.215404 357 tablet_bootstrap.cc:654] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.218954 351 tablet_bootstrap.cc:492] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:16.219436 351 ts_tablet_manager.cc:1397] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.013s
I20260806 04:00:16.226727 352 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:16.227236 352 ts_tablet_manager.cc:1397] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.005s
I20260806 04:00:16.231256 352 raft_consensus.cc:348] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.231940 352 raft_consensus.cc:374] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.232228 352 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.232702 590 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:16.232792 352 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.234565 352 ts_tablet_manager.cc:1428] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:00:16.235262 352 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:16.240386 357 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:16.240872 357 ts_tablet_manager.cc:1397] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.003s
I20260806 04:00:16.241201 351 raft_consensus.cc:348] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.241884 351 raft_consensus.cc:374] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.242159 351 raft_consensus.cc:729] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.242951 351 consensus_queue.cc:260] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.251714 351 ts_tablet_manager.cc:1428] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.032s user 0.006s sys 0.002s
I20260806 04:00:16.252671 351 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:16.255277 357 raft_consensus.cc:348] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.256095 357 raft_consensus.cc:374] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.256379 357 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.257021 357 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.264207 357 ts_tablet_manager.cc:1428] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.023s user 0.006s sys 0.002s
I20260806 04:00:16.265156 357 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:16.274554 352 tablet_bootstrap.cc:654] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.275975 590 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:16.300206 590 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.300901 590 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.304231 590 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.311215 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.312206 351 tablet_bootstrap.cc:654] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.312333 32474 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:16.315801 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.322471 357 tablet_bootstrap.cc:654] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.331069 352 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:16.331542 352 ts_tablet_manager.cc:1397] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.096s user 0.021s sys 0.004s
I20260806 04:00:16.330799 32553 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:16.334708 352 raft_consensus.cc:348] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.335448 352 raft_consensus.cc:374] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.335706 352 raft_consensus.cc:729] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.335755 590 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.336351 590 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.336431 352 consensus_queue.cc:260] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.336875 590 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:16.348596 32553 leader_election.cc:341] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.350797 352 ts_tablet_manager.cc:1428] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20260806 04:00:16.351935 352 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:16.357206 590 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.360431 590 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.368716 380 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:16.370091 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:16.370802 32475 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.372218 380 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.372779 380 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.376783 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.377728 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:16.379595 32428 leader_election.cc:341] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.381289 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:00:16.383543 32427 leader_election.cc:341] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.384373 32427 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:00:16.384670 32553 leader_election.cc:341] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.387984 450 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:16.378882 380 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.389931 660 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:00:16.391727 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:16.392692 660 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.398417 32475 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:16.399441 352 tablet_bootstrap.cc:654] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.408880 351 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:16.409368 351 ts_tablet_manager.cc:1397] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.157s user 0.012s sys 0.055s
I20260806 04:00:16.406297 660 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.412674 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.413538 32475 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:16.407802 32553 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:16.421442 590 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:16.421957 590 raft_consensus.cc:686] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:16.423555 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.424238 590 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.425570 454 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 04:00:16.428697 32553 leader_election.cc:341] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.432212 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:16.416723 351 raft_consensus.cc:348] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.433076 351 raft_consensus.cc:374] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.433270 492 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.433811 492 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.434150 492 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.442773 352 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:16.443275 352 ts_tablet_manager.cc:1397] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.014s
I20260806 04:00:16.443521 492 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.433390 351 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.446875 583 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.446483 351 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.447484 583 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.449177 351 ts_tablet_manager.cc:1428] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.039s user 0.004s sys 0.004s
I20260806 04:00:16.450335 351 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:16.451071 492 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.417730 357 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:16.451885 357 ts_tablet_manager.cc:1397] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.187s user 0.016s sys 0.008s
I20260806 04:00:16.451597 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.452430 32615 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:16.455257 583 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.455096 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:16.456615 32682 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:16.471288 583 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.472903 583 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.473345 583 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.473750 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.474634 32475 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:16.475992 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:16.476868 32474 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.478921 352 raft_consensus.cc:348] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.479692 352 raft_consensus.cc:374] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.479954 352 raft_consensus.cc:729] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.483476 352 consensus_queue.cc:260] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.485747 32321 catalog_manager.cc:5495] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
W20260806 04:00:16.493899 32553 leader_election.cc:341] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.494670 357 raft_consensus.cc:348] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.495357 357 raft_consensus.cc:374] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.495612 357 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.496209 357 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.489050 351 tablet_bootstrap.cc:654] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.496734 352 ts_tablet_manager.cc:1428] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.053s user 0.005s sys 0.003s
I20260806 04:00:16.499025 352 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:16.501972 357 ts_tablet_manager.cc:1428] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20260806 04:00:16.502156 583 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.505030 357 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:16.507504 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:16.508133 32475 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.513689 32474 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:16.515115 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:16.516275 590 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:16.516952 590 raft_consensus.cc:686] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:16.518107 590 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.519373 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:16.520053 32615 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.521649 583 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.526569 32475 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:16.527812 32683 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:16.528792 583 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:16.529428 583 raft_consensus.cc:686] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:16.531934 583 consensus_queue.cc:237] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.540817 357 tablet_bootstrap.cc:654] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.540952 351 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:16.541424 351 ts_tablet_manager.cc:1397] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.014s
I20260806 04:00:16.537539 32615 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:16.544564 352 tablet_bootstrap.cc:654] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.545069 477 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.547289 477 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.545455 351 raft_consensus.cc:348] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.548118 351 raft_consensus.cc:374] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.548413 351 raft_consensus.cc:729] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.549242 351 consensus_queue.cc:260] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.551281 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.551769 351 ts_tablet_manager.cc:1428] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260806 04:00:16.552793 351 tablet_bootstrap.cc:492] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:16.553293 477 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.551282 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.560177 32475 raft_consensus.cc:2379] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:16.560359 351 tablet_bootstrap.cc:654] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.560007 32321 catalog_manager.cc:5495] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:16.561468 32683 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:16.563032 477 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.575464 32326 catalog_manager.cc:5495] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:16.577908 477 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:16.586745 583 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:16.589351 380 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.590039 380 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.594193 380 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.597910 32746 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.596068 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.597592 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:16.606289 32427 leader_election.cc:341] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.607146 583 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:00:16.612774 32428 leader_election.cc:341] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.613438 32428 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:16.619722 380 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:16.621971 718 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.622728 590 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:16.622797 718 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.627693 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.632254 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.633216 32474 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:16.635777 718 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.638222 352 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:16.638854 352 ts_tablet_manager.cc:1397] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.140s user 0.035s sys 0.017s
I20260806 04:00:16.640542 32683 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:16.640708 380 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 04:00:16.653825 351 tablet_bootstrap.cc:492] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:16.654296 351 ts_tablet_manager.cc:1397] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.102s user 0.018s sys 0.034s
I20260806 04:00:16.657089 454 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.657775 454 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.659237 357 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:16.659693 357 ts_tablet_manager.cc:1397] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.155s user 0.008s sys 0.026s
I20260806 04:00:16.661808 590 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:16.662575 357 raft_consensus.cc:348] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.663271 357 raft_consensus.cc:374] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.663516 357 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.652585 718 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:00:16.664638 357 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.668778 454 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.669281 357 ts_tablet_manager.cc:1428] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 04:00:16.673647 357 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:16.676687 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.679061 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.679924 32608 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
W20260806 04:00:16.679095 32428 leader_election.cc:341] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.687131 352 raft_consensus.cc:348] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.689211 351 raft_consensus.cc:348] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.689754 352 raft_consensus.cc:374] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.689991 351 raft_consensus.cc:374] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.690047 352 raft_consensus.cc:729] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.690337 351 raft_consensus.cc:729] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.691040 351 consensus_queue.cc:260] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.692255 357 tablet_bootstrap.cc:654] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.695106 352 consensus_queue.cc:260] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.697160 352 ts_tablet_manager.cc:1428] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.058s user 0.004s sys 0.003s
I20260806 04:00:16.697011 32427 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:16.698524 352 tablet_bootstrap.cc:492] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:16.699757 380 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.704519 380 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.704876 380 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.710911 718 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:00:16.711382 718 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:16.723932 352 tablet_bootstrap.cc:654] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.709976 351 ts_tablet_manager.cc:1428] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.055s user 0.003s sys 0.004s
I20260806 04:00:16.730410 351 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:16.748667 380 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.754554 352 tablet_bootstrap.cc:492] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:16.754861 357 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:16.755043 352 ts_tablet_manager.cc:1397] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.057s user 0.019s sys 0.009s
I20260806 04:00:16.755326 357 ts_tablet_manager.cc:1397] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.085s user 0.013s sys 0.006s
I20260806 04:00:16.742914 718 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.761621 675 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:16.765422 357 raft_consensus.cc:348] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.766095 357 raft_consensus.cc:374] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.766338 357 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.766927 357 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.768689 357 ts_tablet_manager.cc:1428] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20260806 04:00:16.769531 357 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:16.770269 545 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.770782 545 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.773996 351 tablet_bootstrap.cc:654] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.780444 352 raft_consensus.cc:348] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.781220 352 raft_consensus.cc:374] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.781515 352 raft_consensus.cc:729] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.784358 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:16.785081 32608 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.782330 352 consensus_queue.cc:260] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.792186 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:16.792397 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:16.793134 660 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.788722 718 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.793612 660 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.804858 352 ts_tablet_manager.cc:1428] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.049s user 0.009s sys 0.000s
I20260806 04:00:16.795672 545 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.800237 380 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.810014 357 tablet_bootstrap.cc:654] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.818249 660 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.809820 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.821581 32605 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:16.819360 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:16.824950 545 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:00:16.825963 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.829351 32682 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:16.842602 352 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:16.853247 545 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.853847 545 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.854269 545 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:16.860306 32683 leader_election.cc:341] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.865113 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.869257 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:16.876695 32608 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:16.882194 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.886693 32474 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:16.883322 352 tablet_bootstrap.cc:654] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.893765 32427 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:16.894964 454 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:16.895694 454 raft_consensus.cc:686] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:16.896692 454 consensus_queue.cc:237] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.906116 545 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:16.906960 32428 leader_election.cc:341] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.912909 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:00:16.908972 32553 leader_election.cc:341] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.914767 357 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
W20260806 04:00:16.915091 32683 leader_election.cc:341] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.915251 357 ts_tablet_manager.cc:1397] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.146s user 0.022s sys 0.025s
I20260806 04:00:16.915668 352 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:16.917331 352 ts_tablet_manager.cc:1397] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.007s
I20260806 04:00:16.918289 357 raft_consensus.cc:348] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.916558 32553 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:16.919023 357 raft_consensus.cc:374] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.919332 357 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.919559 660 raft_consensus.cc:2781] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.920010 660 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.920590 660 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.926667 352 raft_consensus.cc:348] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.927953 492 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:16.928362 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:16.929714 32605 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.930855 380 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.931466 380 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.919951 357 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.928923 352 raft_consensus.cc:374] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.928977 351 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:16.934120 352 raft_consensus.cc:729] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.934402 351 ts_tablet_manager.cc:1397] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.204s user 0.013s sys 0.021s
I20260806 04:00:16.935935 352 consensus_queue.cc:260] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.937071 357 ts_tablet_manager.cc:1428] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260806 04:00:16.938567 357 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:16.940441 380 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.934005 545 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.940606 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.945995 32605 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:16.946836 583 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.948910 583 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:16.949607 32427 leader_election.cc:341] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.958904 583 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:16.959664 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:16.960278 352 ts_tablet_manager.cc:1428] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.043s user 0.010s sys 0.000s
I20260806 04:00:16.961197 352 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:16.961968 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.963498 32475 raft_consensus.cc:2379] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:16.964677 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.966039 351 raft_consensus.cc:348] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.966698 351 raft_consensus.cc:374] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:16.967044 351 raft_consensus.cc:729] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:16.967507 351 consensus_queue.cc:260] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.965735 590 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:16.969108 32682 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:16.969622 351 ts_tablet_manager.cc:1428] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.035s user 0.003s sys 0.003s
I20260806 04:00:16.969062 590 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.970701 583 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:16.971710 32683 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:16.971673 590 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.972579 718 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.973855 351 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:16.973152 32324 catalog_manager.cc:5495] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:16.980322 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:16.971545 583 raft_consensus.cc:686] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:16.981215 32475 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:16.983574 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:16.981869 583 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:16.989883 32428 leader_election.cc:341] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:16.989559 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:16.990698 357 tablet_bootstrap.cc:654] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.991990 492 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:16.992530 492 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:16.992986 492 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:16.979818 660 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:16.996137 352 tablet_bootstrap.cc:654] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:16.997109 718 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:17.000730 660 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:16.991390 32428 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:17.004041 454 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:17.005964 492 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:17.012048 32553 leader_election.cc:341] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.017746 351 tablet_bootstrap.cc:654] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.013628 492 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.008895 660 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260806 04:00:17.036573 590 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:17.039775 357 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:17.040269 357 ts_tablet_manager.cc:1397] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.102s user 0.014s sys 0.008s
I20260806 04:00:17.041883 32321 catalog_manager.cc:5495] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:17.044356 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:17.045183 32475 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.051455 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:17.051580 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:17.052153 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:17.052537 32478 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.044364 357 raft_consensus.cc:348] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.053365 357 raft_consensus.cc:374] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.053630 357 raft_consensus.cc:729] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.054345 357 consensus_queue.cc:260] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.056694 357 ts_tablet_manager.cc:1428] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20260806 04:00:17.057523 357 tablet_bootstrap.cc:492] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:17.061635 32478 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:17.062513 32746 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.065407 32475 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
W20260806 04:00:17.069703 32553 leader_election.cc:341] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.074990 357 tablet_bootstrap.cc:654] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.077792 32746 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:17.096843 351 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:17.098058 351 ts_tablet_manager.cc:1397] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.009s
I20260806 04:00:17.100692 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:17.103873 590 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:17.104445 454 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:17.105127 454 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.105150 590 raft_consensus.cc:686] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:17.107424 590 consensus_queue.cc:237] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.108306 32553 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:17.108888 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:17.109231 660 raft_consensus.cc:2781] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:17.109727 32605 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:17.112928 351 raft_consensus.cc:348] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.113710 351 raft_consensus.cc:374] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.113963 351 raft_consensus.cc:729] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.117815 32427 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:17.119167 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:17.118963 736 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:17.120343 736 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:17.120894 736 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:17.121203 32428 leader_election.cc:341] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.121590 583 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:17.126844 352 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:17.127353 352 ts_tablet_manager.cc:1397] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.166s user 0.003s sys 0.043s
I20260806 04:00:17.115411 351 consensus_queue.cc:260] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.122059 454 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.132555 352 raft_consensus.cc:348] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.133273 352 raft_consensus.cc:374] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.133546 352 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.138882 351 ts_tablet_manager.cc:1428] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260806 04:00:17.140172 351 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:17.141700 352 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.145501 583 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:17.153400 352 ts_tablet_manager.cc:1428] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.026s user 0.005s sys 0.003s
I20260806 04:00:17.154522 352 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:17.161863 660 raft_consensus.cc:686] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:17.166848 357 tablet_bootstrap.cc:492] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:17.167600 357 ts_tablet_manager.cc:1397] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.110s user 0.023s sys 0.014s
I20260806 04:00:17.162856 660 consensus_queue.cc:237] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.205886 351 tablet_bootstrap.cc:654] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.206812 736 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.228456 352 tablet_bootstrap.cc:654] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.230166 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:17.231122 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:17.231824 736 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.176229 753 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
W20260806 04:00:17.232913 32428 leader_election.cc:341] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.232949 32615 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.236937 357 raft_consensus.cc:348] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.238075 357 raft_consensus.cc:374] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.238384 357 raft_consensus.cc:729] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.239184 357 consensus_queue.cc:260] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.240034 718 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:17.240640 718 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.243700 718 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:17.243935 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:17.244573 32475 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:17.244861 32615 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:17.246249 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:17.247757 32683 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:17.249207 357 ts_tablet_manager.cc:1428] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.081s user 0.005s sys 0.002s
I20260806 04:00:17.252028 583 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:17.254916 583 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:17.255304 583 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.253509 357 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
W20260806 04:00:17.251003 32682 leader_election.cc:341] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.263860 32427 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:17.264873 736 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:17.265648 736 raft_consensus.cc:686] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:17.266743 736 consensus_queue.cc:237] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:17.249928 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 66169.5us
I20260806 04:00:17.290704 583 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.306288 357 tablet_bootstrap.cc:654] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.308946 351 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:17.309831 351 ts_tablet_manager.cc:1397] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.170s user 0.021s sys 0.013s
I20260806 04:00:17.347772 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:17.348584 32477 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.350175 590 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:17.350739 32749 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
W20260806 04:00:17.351262 32428 consensus_peers.cc:487] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 -> Peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Couldn't send request to peer f7c202b562ce482db283463d7f02e136. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:17.352178 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:17.352994 352 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:17.353734 352 ts_tablet_manager.cc:1397] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.199s user 0.018s sys 0.016s
I20260806 04:00:17.350761 590 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.354444 583 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:00:17.355533 32682 leader_election.cc:341] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.360707 660 consensus_queue.cc:1046] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:17.361938 757 raft_consensus.cc:3032] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:17.362352 757 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:17.363013 590 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.363425 757 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.365196 757 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:17.368469 32326 catalog_manager.cc:5495] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:17.380162 351 raft_consensus.cc:348] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.381079 351 raft_consensus.cc:374] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.381348 351 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.391450 764 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:17.399674 764 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.399056 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:17.409520 351 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.409770 32475 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:17.367883 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:17.411949 352 raft_consensus.cc:348] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.412984 352 raft_consensus.cc:374] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.412941 32553 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:17.413468 352 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.415563 352 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.410681 32321 catalog_manager.cc:5495] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:17.417479 32321 catalog_manager.cc:5495] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:17.421972 351 ts_tablet_manager.cc:1428] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.111s user 0.015s sys 0.001s
I20260806 04:00:17.423041 351 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
W20260806 04:00:17.425163 32553 leader_election.cc:341] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.426540 352 ts_tablet_manager.cc:1428] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.072s user 0.013s sys 0.003s
I20260806 04:00:17.427541 352 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:17.417626 660 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:17.429680 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:17.432425 32475 raft_consensus.cc:2379] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:17.428428 660 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:17.432919 660 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.433986 764 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:17.434092 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:17.437120 32683 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:17.445449 32477 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:17.447813 764 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.449397 357 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:17.449242 32683 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:17.449849 357 ts_tablet_manager.cc:1397] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.200s user 0.027s sys 0.027s
I20260806 04:00:17.453540 583 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:17.454954 583 raft_consensus.cc:686] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:17.456174 583 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.461701 718 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:17.478338 660 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.496881 352 tablet_bootstrap.cc:654] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.491655 357 raft_consensus.cc:348] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.503093 660 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.504272 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:17.506925 32324 catalog_manager.cc:5495] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:17.499905 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:17.512794 32477 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.504536 357 raft_consensus.cc:374] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.513278 357 raft_consensus.cc:729] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.513971 357 consensus_queue.cc:260] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.516235 32746 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.518808 351 tablet_bootstrap.cc:654] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.520730 769 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260806 04:00:17.544591 357 ts_tablet_manager.cc:1428] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.094s user 0.005s sys 0.005s
I20260806 04:00:17.532838 757 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
W20260806 04:00:17.566848 32683 consensus_peers.cc:487] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 -> Peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): Couldn't send request to peer ae0b3e43e7a64f789e51018fd5518d23. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:17.575501 32746 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:17.581728 764 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:17.586058 357 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
W20260806 04:00:17.588536 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:17.590638 352 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:17.591120 352 ts_tablet_manager.cc:1397] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.164s user 0.016s sys 0.016s
I20260806 04:00:17.590797 752 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:17.592451 351 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:17.596412 351 ts_tablet_manager.cc:1397] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.174s user 0.014s sys 0.010s
I20260806 04:00:17.600641 352 raft_consensus.cc:348] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.601542 352 raft_consensus.cc:374] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.601933 352 raft_consensus.cc:729] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.603513 32477 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:17.604038 352 consensus_queue.cc:260] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.606704 352 ts_tablet_manager.cc:1428] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260806 04:00:17.607498 32553 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:17.607717 352 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:17.608407 752 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:17.609009 752 raft_consensus.cc:686] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:17.611065 752 consensus_queue.cc:237] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.625432 357 tablet_bootstrap.cc:654] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.701848 351 raft_consensus.cc:348] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.703907 351 raft_consensus.cc:374] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.704303 351 raft_consensus.cc:729] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.707522 351 consensus_queue.cc:260] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.708421 789 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:17.709568 789 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.719581 32321 catalog_manager.cc:5495] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
W20260806 04:00:17.731173 32428 consensus_peers.cc:487] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 -> Peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Couldn't send request to peer f7c202b562ce482db283463d7f02e136. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:17.734889 352 tablet_bootstrap.cc:654] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.731752 757 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:17.743719 351 ts_tablet_manager.cc:1428] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.147s user 0.011s sys 0.007s
I20260806 04:00:17.747738 351 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:17.752442 757 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.762809 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:17.763257 757 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.763187 791 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:17.764010 757 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:17.764961 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:17.765756 32605 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:17.763733 32478 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:17.767686 789 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.769263 32553 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:17.773268 752 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:17.773191 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:17.773727 752 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:17.774165 752 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.774889 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:17.773077 32427 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:00:17.782931 32553 leader_election.cc:341] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.787742 357 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:17.789160 732 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:17.802218 357 ts_tablet_manager.cc:1397] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.216s user 0.024s sys 0.010s
I20260806 04:00:17.794835 732 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:17.803838 732 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.806363 357 raft_consensus.cc:348] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.807150 357 raft_consensus.cc:374] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.807418 357 raft_consensus.cc:729] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.824669 752 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.828008 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:17.832408 351 tablet_bootstrap.cc:654] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.808046 357 consensus_queue.cc:260] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.837198 357 ts_tablet_manager.cc:1428] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.034s user 0.004s sys 0.001s
I20260806 04:00:17.838444 357 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:17.839131 752 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.845625 352 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:17.845882 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:17.846187 352 ts_tablet_manager.cc:1397] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.239s user 0.012s sys 0.008s
W20260806 04:00:17.847076 32553 leader_election.cc:341] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.849277 380 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:00:17.847996 732 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.852360 32478 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:17.855581 380 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:17.856021 32553 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:17.856029 380 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.858987 380 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.860067 492 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:00:17.860517 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:17.861143 32615 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:17.861423 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:17.863137 32428 leader_election.cc:341] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.863314 352 raft_consensus.cc:348] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.864148 352 raft_consensus.cc:374] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.864415 352 raft_consensus.cc:729] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.864248 32427 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:17.866159 732 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.866393 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:17.866679 732 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:17.866986 732 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:17.867308 732 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.865074 352 consensus_queue.cc:260] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:17.867965 32428 leader_election.cc:341] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.870087 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:17.870862 32615 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:17.872776 32427 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:17.873862 769 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:00:17.888470 732 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.891062 732 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.892251 352 ts_tablet_manager.cc:1428] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.046s user 0.008s sys 0.001s
I20260806 04:00:17.893538 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:00:17.894649 32682 consensus_peers.cc:487] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 -> Peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Couldn't send request to peer dd7344d85a154539b7f8c484f2a76772. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260806 04:00:17.895586 32428 leader_election.cc:341] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:17.898173 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:17.898772 32605 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.902861 357 tablet_bootstrap.cc:654] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.907716 352 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:17.910842 32605 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:17.912590 32427 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:17.913998 732 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:17.914774 732 raft_consensus.cc:686] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:17.915793 732 consensus_queue.cc:237] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.920277 352 tablet_bootstrap.cc:654] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.928052 351 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:17.928498 357 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:17.928647 351 ts_tablet_manager.cc:1397] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.181s user 0.012s sys 0.022s
I20260806 04:00:17.929003 357 ts_tablet_manager.cc:1397] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.005s
I20260806 04:00:17.932870 351 raft_consensus.cc:348] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.934792 351 raft_consensus.cc:374] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.935202 351 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.939119 351 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.940802 351 ts_tablet_manager.cc:1428] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20260806 04:00:17.942322 351 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:17.948046 351 tablet_bootstrap.cc:654] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.943557 32321 catalog_manager.cc:5495] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:17.954504 357 raft_consensus.cc:348] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.955495 357 raft_consensus.cc:374] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.958326 352 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:17.958777 352 ts_tablet_manager.cc:1397] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.000s
I20260806 04:00:17.955806 357 raft_consensus.cc:729] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.963492 357 consensus_queue.cc:260] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.965238 357 ts_tablet_manager.cc:1428] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.036s user 0.002s sys 0.004s
I20260806 04:00:17.968003 357 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:17.968500 351 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:17.968930 351 ts_tablet_manager.cc:1397] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.004s
I20260806 04:00:17.971516 351 raft_consensus.cc:348] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.972213 351 raft_consensus.cc:374] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.972465 351 raft_consensus.cc:729] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.973512 357 tablet_bootstrap.cc:654] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:17.973083 351 consensus_queue.cc:260] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.975937 351 ts_tablet_manager.cc:1428] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:00:17.978050 351 tablet_bootstrap.cc:492] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:17.978724 352 raft_consensus.cc:348] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.979501 352 raft_consensus.cc:374] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:17.979779 352 raft_consensus.cc:729] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:17.980433 352 consensus_queue.cc:260] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.981103 732 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:17.981570 732 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:17.983600 732 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:17.984959 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:17.985631 32745 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:17.986922 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:17.987517 32605 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:17.988382 32428 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:17.989306 769 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:17.989620 769 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:17.989928 769 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:17.995230 352 ts_tablet_manager.cc:1428] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20260806 04:00:17.996405 352 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:17.999040 769 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.001708 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:18.002269 32605 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.006934 352 tablet_bootstrap.cc:654] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.009588 351 tablet_bootstrap.cc:654] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.011493 32605 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:18.011811 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:18.012589 32748 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.012998 718 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.013458 718 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.016492 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.016604 718 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:18.017468 32478 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:18.017772 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:18.019662 32682 leader_election.cc:341] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.020398 32748 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:18.020522 32683 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:18.021440 32428 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:18.022161 732 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:18.022966 732 raft_consensus.cc:686] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:18.023974 732 consensus_queue.cc:237] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.026953 718 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:18.027362 718 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:18.027694 718 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.031752 769 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.039662 718 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.041496 357 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.041647 718 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:18.043403 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:00:18.044862 32682 leader_election.cc:341] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.043690 357 ts_tablet_manager.cc:1397] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.076s user 0.047s sys 0.010s
I20260806 04:00:18.047734 352 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.048326 352 ts_tablet_manager.cc:1397] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.001s
I20260806 04:00:18.048026 357 raft_consensus.cc:348] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.048801 357 raft_consensus.cc:374] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.049074 357 raft_consensus.cc:729] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.049696 357 consensus_queue.cc:260] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.051049 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:18.051187 352 raft_consensus.cc:348] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.051906 357 ts_tablet_manager.cc:1428] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260806 04:00:18.051981 352 raft_consensus.cc:374] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.052397 352 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.053325 357 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:18.053035 352 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.051640 32475 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.055465 352 ts_tablet_manager.cc:1428] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:00:18.055306 583 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:18.056630 352 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.063374 352 tablet_bootstrap.cc:654] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.066100 357 tablet_bootstrap.cc:654] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.067816 351 tablet_bootstrap.cc:492] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.068408 351 ts_tablet_manager.cc:1397] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.091s user 0.015s sys 0.010s
I20260806 04:00:18.069919 32475 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:18.070911 32683 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:18.071972 718 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:18.072440 718 raft_consensus.cc:686] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:18.073232 718 consensus_queue.cc:237] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.070873 351 raft_consensus.cc:348] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.078711 351 raft_consensus.cc:374] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.078984 351 raft_consensus.cc:729] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.079406 752 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:18.079957 351 consensus_queue.cc:260] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.081337 32321 catalog_manager.cc:5495] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:18.086925 357 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.087481 357 ts_tablet_manager.cc:1397] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.012s
I20260806 04:00:18.090586 351 ts_tablet_manager.cc:1428] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260806 04:00:18.090926 357 raft_consensus.cc:348] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.091766 357 raft_consensus.cc:374] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.092203 357 raft_consensus.cc:729] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.093066 357 consensus_queue.cc:260] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.094568 32326 catalog_manager.cc:5495] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:18.091831 351 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:18.095650 357 ts_tablet_manager.cc:1428] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260806 04:00:18.106667 590 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:18.110855 357 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:18.128435 351 tablet_bootstrap.cc:654] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.165091 357 tablet_bootstrap.cc:654] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.170217 352 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.170666 352 ts_tablet_manager.cc:1397] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.114s user 0.018s sys 0.011s
I20260806 04:00:18.173559 352 raft_consensus.cc:348] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.174271 352 raft_consensus.cc:374] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.174539 352 raft_consensus.cc:729] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.175238 352 consensus_queue.cc:260] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.177433 352 ts_tablet_manager.cc:1428] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:00:18.178896 352 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.185602 352 tablet_bootstrap.cc:654] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.189042 583 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.189559 583 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.191612 583 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:18.193287 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.194173 32605 raft_consensus.cc:2379] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:18.196614 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.197743 352 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.198187 352 ts_tablet_manager.cc:1397] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20260806 04:00:18.198047 32683 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:18.199186 583 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.201138 352 raft_consensus.cc:348] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.201999 352 raft_consensus.cc:374] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.202266 352 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.202932 352 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.204461 352 ts_tablet_manager.cc:1428] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260806 04:00:18.205824 352 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.206729 357 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.207192 357 ts_tablet_manager.cc:1397] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.097s user 0.016s sys 0.008s
I20260806 04:00:18.209702 357 raft_consensus.cc:348] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.210438 357 raft_consensus.cc:374] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.210685 357 raft_consensus.cc:729] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.211440 583 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.211335 357 consensus_queue.cc:260] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.212218 352 tablet_bootstrap.cc:654] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.213515 357 ts_tablet_manager.cc:1428] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:00:18.214478 357 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:18.218621 732 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.219128 732 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.221171 732 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.221366 357 tablet_bootstrap.cc:654] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.222922 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.224421 352 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.224881 352 ts_tablet_manager.cc:1397] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260806 04:00:18.225509 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.226630 351 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.227062 351 ts_tablet_manager.cc:1397] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.136s user 0.014s sys 0.022s
W20260806 04:00:18.226980 32427 leader_election.cc:341] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:18.229614 32428 leader_election.cc:341] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.229998 32428 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.230065 351 raft_consensus.cc:348] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.230760 351 raft_consensus.cc:374] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.231222 351 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.231559 769 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.232203 675 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.231868 351 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.232642 675 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.239463 351 ts_tablet_manager.cc:1428] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260806 04:00:18.241147 352 raft_consensus.cc:348] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.241967 352 raft_consensus.cc:374] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.242266 352 raft_consensus.cc:729] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.242933 352 consensus_queue.cc:260] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.243922 351 tablet_bootstrap.cc:492] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:18.245649 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.246407 352 ts_tablet_manager.cc:1428] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260806 04:00:18.244280 32748 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.246894 590 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.247392 590 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.249214 590 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.250411 357 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.250871 357 ts_tablet_manager.cc:1397] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.004s
I20260806 04:00:18.251701 675 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.253268 357 raft_consensus.cc:348] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.253635 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.253928 357 raft_consensus.cc:374] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.254060 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:18.255709 32427 leader_election.cc:341] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:18.256701 32428 leader_election.cc:341] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.257174 32428 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.255781 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.259907 32553 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.260251 352 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.262316 752 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.265993 352 tablet_bootstrap.cc:654] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.258256 732 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.254222 357 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.275048 357 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.290696 357 ts_tablet_manager.cc:1428] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.040s user 0.005s sys 0.001s
I20260806 04:00:18.292677 357 tablet_bootstrap.cc:492] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:18.297186 351 tablet_bootstrap.cc:654] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.297319 352 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.297796 352 ts_tablet_manager.cc:1397] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.009s
I20260806 04:00:18.299157 357 tablet_bootstrap.cc:654] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.300397 352 raft_consensus.cc:348] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.301115 352 raft_consensus.cc:374] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.301426 352 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.302007 352 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.304414 352 ts_tablet_manager.cc:1428] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260806 04:00:18.305830 352 tablet_bootstrap.cc:492] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.311437 352 tablet_bootstrap.cc:654] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.326517 32748 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.334667 351 tablet_bootstrap.cc:492] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.334779 357 tablet_bootstrap.cc:492] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.335191 351 ts_tablet_manager.cc:1397] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.004s
I20260806 04:00:18.335562 357 ts_tablet_manager.cc:1397] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.032s
I20260806 04:00:18.338680 357 raft_consensus.cc:348] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.339893 357 raft_consensus.cc:374] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.339993 352 tablet_bootstrap.cc:492] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.340262 357 raft_consensus.cc:729] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.340544 352 ts_tablet_manager.cc:1397] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.003s
I20260806 04:00:18.338581 351 raft_consensus.cc:348] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.341037 351 raft_consensus.cc:374] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.341310 351 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.341295 357 consensus_queue.cc:260] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.342214 351 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.342903 352 raft_consensus.cc:348] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.343595 352 raft_consensus.cc:374] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.343851 352 raft_consensus.cc:729] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.344442 352 consensus_queue.cc:260] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.346482 352 ts_tablet_manager.cc:1428] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260806 04:00:18.347481 352 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.348841 583 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.349292 583 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.351004 583 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:18.353238 352 tablet_bootstrap.cc:654] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.353145 675 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:18.353778 357 ts_tablet_manager.cc:1428] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260806 04:00:18.354907 357 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:18.355909 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.356603 32605 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:18.353178 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.358186 32477 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:18.358373 32682 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:18.360201 764 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:18.360569 764 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:18.360908 764 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.364094 590 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.364575 590 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.366479 590 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.368640 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.369313 32745 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:18.370373 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.371106 32478 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:18.371882 32553 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:18.373584 590 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:18.374022 590 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:18.374362 590 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.378410 32605 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.383015 590 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.387241 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:18.387866 32478 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.373577 764 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.399133 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:18.400188 32748 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.407371 357 tablet_bootstrap.cc:654] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.396222 32478 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:18.407843 590 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.408471 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:18.412909 764 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:18.413625 32477 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.416395 32553 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:18.416579 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:18.417173 32610 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.417241 590 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:18.417809 590 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:18.418659 590 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.408815 454 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.420866 454 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.423967 351 ts_tablet_manager.cc:1428] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.083s user 0.006s sys 0.002s
I20260806 04:00:18.423870 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.424983 351 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:18.427551 32477 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:18.428395 732 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:00:18.428524 32427 leader_election.cc:341] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.428987 732 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.430210 454 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.430981 732 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.431233 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.432685 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:18.433369 32428 leader_election.cc:341] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.433784 32428 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:00:18.434166 32427 leader_election.cc:341] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.434465 32610 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:18.434706 732 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.411720 32748 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:18.435665 32682 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:18.436538 764 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:18.437008 764 raft_consensus.cc:686] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:18.437912 764 consensus_queue.cc:237] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.435678 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:18.443923 32428 leader_election.cc:341] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.444411 32428 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.445493 732 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.448637 351 tablet_bootstrap.cc:654] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.455487 352 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.460981 352 ts_tablet_manager.cc:1397] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.114s user 0.042s sys 0.038s
I20260806 04:00:18.463644 352 raft_consensus.cc:348] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.464156 352 raft_consensus.cc:374] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.464346 352 raft_consensus.cc:729] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.464855 352 consensus_queue.cc:260] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.466020 352 ts_tablet_manager.cc:1428] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:00:18.467046 352 tablet_bootstrap.cc:492] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.466621 32324 catalog_manager.cc:5495] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:18.472576 352 tablet_bootstrap.cc:654] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.479463 357 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.480528 357 ts_tablet_manager.cc:1397] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.126s user 0.015s sys 0.003s
I20260806 04:00:18.484972 454 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:18.485906 352 tablet_bootstrap.cc:492] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.486374 352 ts_tablet_manager.cc:1397] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20260806 04:00:18.488905 352 raft_consensus.cc:348] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.489594 352 raft_consensus.cc:374] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.489845 352 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.490445 352 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.498358 352 ts_tablet_manager.cc:1428] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260806 04:00:18.499336 352 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.500743 454 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:18.494002 351 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.502167 32324 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:18.482954 357 raft_consensus.cc:348] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.504601 352 tablet_bootstrap.cc:654] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.504781 357 raft_consensus.cc:374] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.505127 357 raft_consensus.cc:729] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.506816 454 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.506678 357 consensus_queue.cc:260] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.504736 351 ts_tablet_manager.cc:1397] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.003s
I20260806 04:00:18.507313 454 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.509737 454 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.509863 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.510663 351 raft_consensus.cc:348] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.511121 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.511451 32615 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:18.512547 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:18.513137 769 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 04:00:18.513311 32428 leader_election.cc:341] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.513502 769 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:18.513825 769 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.514081 351 raft_consensus.cc:374] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.514343 351 raft_consensus.cc:729] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.515074 351 consensus_queue.cc:260] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.517252 351 ts_tablet_manager.cc:1428] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 04:00:18.518184 351 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:18.520282 769 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.522766 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:18.523375 32615 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.529640 769 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.531673 454 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.532157 454 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.532485 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:00:18.533715 32428 leader_election.cc:341] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.533763 32615 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:18.534652 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.535565 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.536203 732 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:18.536736 454 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.536689 732 raft_consensus.cc:686] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:18.537714 732 consensus_queue.cc:237] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:18.538569 32682 consensus_peers.cc:487] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 -> Peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Couldn't send request to peer dd7344d85a154539b7f8c484f2a76772. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:18.538529 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.539973 352 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.540443 352 ts_tablet_manager.cc:1397] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.000s
I20260806 04:00:18.540409 32428 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.541309 351 tablet_bootstrap.cc:654] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.541342 675 raft_consensus.cc:2726] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.542910 352 raft_consensus.cc:348] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.543602 352 raft_consensus.cc:374] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.543859 352 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.544517 352 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.547585 352 ts_tablet_manager.cc:1428] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:00:18.548628 352 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.549710 357 ts_tablet_manager.cc:1428] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.069s user 0.004s sys 0.002s
I20260806 04:00:18.553785 352 tablet_bootstrap.cc:654] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.553130 32326 catalog_manager.cc:5495] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:18.555326 357 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:18.565009 357 tablet_bootstrap.cc:654] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.571627 675 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:18.581022 352 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.581496 352 ts_tablet_manager.cc:1397] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.012s
I20260806 04:00:18.582250 764 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:18.583995 352 raft_consensus.cc:348] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.584683 352 raft_consensus.cc:374] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.600234 351 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.585074 352 raft_consensus.cc:729] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.600730 351 ts_tablet_manager.cc:1397] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.083s user 0.025s sys 0.004s
I20260806 04:00:18.601286 352 consensus_queue.cc:260] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.603302 352 ts_tablet_manager.cc:1428] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260806 04:00:18.603221 351 raft_consensus.cc:348] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.604043 351 raft_consensus.cc:374] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.604388 351 raft_consensus.cc:729] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.604337 352 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.612021 357 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.616110 357 ts_tablet_manager.cc:1397] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.007s
I20260806 04:00:18.605074 351 consensus_queue.cc:260] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.618989 351 ts_tablet_manager.cc:1428] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260806 04:00:18.618690 357 raft_consensus.cc:348] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.619405 357 raft_consensus.cc:374] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.619693 357 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.619916 351 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:18.620328 357 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.622408 357 ts_tablet_manager.cc:1428] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:00:18.625427 351 tablet_bootstrap.cc:654] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.629851 454 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:00:18.630462 454 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.632514 454 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.633592 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.634038 454 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.634442 32748 raft_consensus.cc:2379] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 2: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:00:18.635293 675 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.635748 675 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.636318 352 tablet_bootstrap.cc:654] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.636987 351 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.637575 351 ts_tablet_manager.cc:1397] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20260806 04:00:18.637768 675 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.638574 795 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.638655 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.639081 795 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.641100 795 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.639392 32605 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:18.634526 454 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.641683 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.642346 32610 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:18.643101 454 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.643641 351 raft_consensus.cc:348] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.644289 351 raft_consensus.cc:374] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.644531 351 raft_consensus.cc:729] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.645435 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.645242 351 consensus_queue.cc:260] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.647358 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.647650 351 ts_tablet_manager.cc:1428] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260806 04:00:18.648115 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:18.648437 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.648891 351 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:18.649051 675 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:00:18.649385 32427 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:18.649544 675 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:00:18.650058 675 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:18.650094 454 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:18.650574 454 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:18.650909 454 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.651475 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.652036 32746 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:18.653622 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.656910 675 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.657675 32478 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.662088 454 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.667013 32477 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:18.658517 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:18.658849 357 tablet_bootstrap.cc:492] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:18.659961 351 tablet_bootstrap.cc:654] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
W20260806 04:00:18.660972 32428 leader_election.cc:341] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.674176 351 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.671171 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:18.674664 351 ts_tablet_manager.cc:1397] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.002s
I20260806 04:00:18.674953 32746 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.677358 351 raft_consensus.cc:348] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.678285 351 raft_consensus.cc:374] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.678573 351 raft_consensus.cc:729] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.679150 351 consensus_queue.cc:260] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.674680 795 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:18.680500 795 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:18.680830 795 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.681135 357 tablet_bootstrap.cc:654] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.668062 454 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.681406 351 ts_tablet_manager.cc:1428] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 04:00:18.684247 351 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
W20260806 04:00:18.686941 32427 leader_election.cc:341] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.663367 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.690338 351 tablet_bootstrap.cc:654] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.692929 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:18.693480 675 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.703775 357 tablet_bootstrap.cc:492] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.704357 357 ts_tablet_manager.cc:1397] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.004s
I20260806 04:00:18.706686 357 raft_consensus.cc:348] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.713505 357 raft_consensus.cc:374] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.715967 357 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.716776 357 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.719376 357 ts_tablet_manager.cc:1428] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260806 04:00:18.720350 357 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:18.693711 32748 raft_consensus.cc:2379] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 2: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:00:18.710628 32427 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.715605 352 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.723325 352 ts_tablet_manager.cc:1397] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.119s user 0.034s sys 0.030s
I20260806 04:00:18.723393 675 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 04:00:18.723496 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 57871us
I20260806 04:00:18.726596 352 raft_consensus.cc:348] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.727944 352 raft_consensus.cc:374] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.728281 352 raft_consensus.cc:729] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.728965 352 consensus_queue.cc:260] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.730602 352 ts_tablet_manager.cc:1428] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:00:18.731585 352 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.733656 32746 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:18.735873 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:18.736776 795 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.737926 32605 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:18.739270 357 tablet_bootstrap.cc:654] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.741138 352 tablet_bootstrap.cc:654] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.745141 454 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.745092 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:18.745661 454 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.747394 454 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.749737 795 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.751789 352 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.752653 352 ts_tablet_manager.cc:1397] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20260806 04:00:18.753324 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:18.754012 32478 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:18.755558 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:18.755811 351 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.756254 32746 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:18.756359 351 ts_tablet_manager.cc:1397] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.072s user 0.021s sys 0.036s
I20260806 04:00:18.756036 352 raft_consensus.cc:348] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.757071 352 raft_consensus.cc:374] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.757354 352 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.757663 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.758184 764 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.757961 352 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.758610 764 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.759629 352 ts_tablet_manager.cc:1428] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260806 04:00:18.759317 351 raft_consensus.cc:348] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.760133 351 raft_consensus.cc:374] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.760462 351 raft_consensus.cc:729] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.760731 352 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.761485 795 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.761184 351 consensus_queue.cc:260] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.762773 351 ts_tablet_manager.cc:1428] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:00:18.764333 351 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:18.766829 352 tablet_bootstrap.cc:654] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.769639 351 tablet_bootstrap.cc:654] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.774895 764 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:18.778021 352 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.778510 352 ts_tablet_manager.cc:1397] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260806 04:00:18.778399 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.779964 32428 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:18.780740 454 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:18.781220 454 raft_consensus.cc:686] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:18.781798 352 raft_consensus.cc:348] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.782058 454 consensus_queue.cc:237] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.782649 352 raft_consensus.cc:374] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.782971 352 raft_consensus.cc:729] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.783674 352 consensus_queue.cc:260] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.785537 352 ts_tablet_manager.cc:1428] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 04:00:18.786593 352 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.787022 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.787969 32478 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:18.794678 352 tablet_bootstrap.cc:654] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.799386 675 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.799860 675 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.801734 675 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.804154 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.805110 32610 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:18.806929 352 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.807359 352 ts_tablet_manager.cc:1397] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.005s
I20260806 04:00:18.808121 795 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:18.809680 352 raft_consensus.cc:348] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.810346 352 raft_consensus.cc:374] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.810587 352 raft_consensus.cc:729] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.811131 352 consensus_queue.cc:260] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.812930 352 ts_tablet_manager.cc:1428] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:00:18.813861 352 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
W20260806 04:00:18.817117 32428 leader_election.cc:341] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.819276 352 tablet_bootstrap.cc:654] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.828454 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.829146 32615 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:18.830755 352 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.831264 352 ts_tablet_manager.cc:1397] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260806 04:00:18.833143 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.834173 352 raft_consensus.cc:348] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.834321 32324 catalog_manager.cc:5495] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:18.835301 352 raft_consensus.cc:374] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.835750 352 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.836423 352 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.838222 352 ts_tablet_manager.cc:1428] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:00:18.839211 352 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.844903 352 tablet_bootstrap.cc:654] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.836378 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.862097 732 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.862751 732 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.871317 32682 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:18.872407 732 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.877345 764 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.879014 352 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:18.879485 352 ts_tablet_manager.cc:1397] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.025s
I20260806 04:00:18.882067 352 raft_consensus.cc:348] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.882874 352 raft_consensus.cc:374] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.883198 352 raft_consensus.cc:729] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.883361 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.883970 352 consensus_queue.cc:260] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.889151 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:18.889926 32428 leader_election.cc:341] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.900977 769 raft_consensus.cc:2781] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:18.901341 769 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:18.889804 752 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:18.901664 769 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:18.902153 32428 leader_election.cc:341] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.895613 357 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:18.903801 357 ts_tablet_manager.cc:1397] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.184s user 0.041s sys 0.028s
I20260806 04:00:18.906203 357 raft_consensus.cc:348] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.907346 769 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.909276 769 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.893000 675 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.907791 357 raft_consensus.cc:374] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.912578 357 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:18.899744 351 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:18.913561 351 ts_tablet_manager.cc:1397] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.149s user 0.050s sys 0.014s
I20260806 04:00:18.902297 352 ts_tablet_manager.cc:1428] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.023s user 0.003s sys 0.004s
I20260806 04:00:18.913385 357 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.914777 352 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:18.912640 675 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.916132 357 ts_tablet_manager.cc:1428] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 04:00:18.917096 357 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
W20260806 04:00:18.917053 32427 leader_election.cc:341] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.919742 32427 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.922009 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.926270 32605 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:18.926424 583 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.926896 583 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.917038 675 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.934244 583 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:18.935356 894 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.915992 764 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.934369 351 raft_consensus.cc:348] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.937381 351 raft_consensus.cc:374] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:18.937700 351 raft_consensus.cc:729] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
W20260806 04:00:18.938185 32427 leader_election.cc:341] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:18.955716 32427 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.954135 764 consensus_queue.cc:1035] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:18.938838 352 tablet_bootstrap.cc:654] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.939214 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:18.958617 32615 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.959340 901 consensus_queue.cc:1035] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:18.942744 357 tablet_bootstrap.cc:654] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:18.945011 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:18.950456 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:18.950866 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.971087 32608 raft_consensus.cc:2379] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:18.952013 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.958117 899 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:18.958206 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:18.977100 795 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.977610 795 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.940445 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:18.979180 899 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:18.979614 795 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.964039 32748 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.964435 32615 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:18.955706 351 consensus_queue.cc:260] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.982657 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:18.983316 454 raft_consensus.cc:2781] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:18.983445 351 ts_tablet_manager.cc:1428] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.070s user 0.005s sys 0.000s
I20260806 04:00:18.983783 454 raft_consensus.cc:686] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:18.967177 900 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:18.984495 900 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.984669 454 consensus_queue.cc:237] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.982991 32605 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:18.986431 900 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:18.989341 351 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:18.979684 899 raft_consensus.cc:686] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:18.993099 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:18.993335 899 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:18.993837 32477 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:18.994791 32553 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:18.997414 795 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:18.997766 795 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:18.998080 795 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:18.999912 351 tablet_bootstrap.cc:654] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.000048 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.004850 32427 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.005650 898 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:19.005836 32605 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.006009 898 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:00:19.005892 32427 leader_election.cc:341] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.006364 898 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.008481 583 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.008868 900 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:19.010216 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.008890 583 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.012164 583 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:19.012990 898 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:19.013561 32553 leader_election.cc:341] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.014398 32683 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:19.015067 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.015641 898 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.019709 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.020416 32610 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:00:19.021664 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:19.022589 764 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:00:19.022744 583 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.023010 764 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.023558 764 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:19.026582 351 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:19.027055 351 ts_tablet_manager.cc:1397] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.003s
I20260806 04:00:19.030213 764 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.031252 351 raft_consensus.cc:348] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.031909 351 raft_consensus.cc:374] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.032195 351 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.033177 764 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:19.035746 351 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.037029 898 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.044036 898 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.044919 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:19.045518 32608 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:19.045949 898 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.042019 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.043569 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:19.050414 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.051028 32610 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:19.042768 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:19.052314 32615 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.053164 32608 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:00:19.053243 32427 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.054024 454 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:19.054338 454 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.054646 454 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.055723 795 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.056794 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:19.064369 352 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:19.064858 352 ts_tablet_manager.cc:1397] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.150s user 0.035s sys 0.017s
I20260806 04:00:19.060971 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:19.064668 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:19.065728 764 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:00:19.066954 32615 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:19.066177 32477 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.067924 764 raft_consensus.cc:686] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:19.067911 454 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.069316 32427 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.069948 454 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:00:19.059473 32428 leader_election.cc:341] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.070333 732 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:19.070772 732 raft_consensus.cc:686] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:00:19.071357 32428 leader_election.cc:341] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.071727 732 consensus_queue.cc:237] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:19.073020 32428 leader_election.cc:341] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.072160 795 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.073813 32428 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:19.075013 900 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.069005 764 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.079084 352 raft_consensus.cc:348] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.079731 352 raft_consensus.cc:374] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.079802 351 ts_tablet_manager.cc:1428] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.052s user 0.007s sys 0.000s
I20260806 04:00:19.080029 352 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.080435 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:19.080958 32615 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.080639 352 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.082336 351 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:19.082685 352 ts_tablet_manager.cc:1428] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20260806 04:00:19.082834 357 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:19.083374 357 ts_tablet_manager.cc:1397] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.166s user 0.015s sys 0.011s
I20260806 04:00:19.083613 352 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:19.085556 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:19.085824 357 raft_consensus.cc:348] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.086462 357 raft_consensus.cc:374] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.086721 357 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.089893 32477 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:19.091022 32553 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.091756 752 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:19.092226 752 raft_consensus.cc:686] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:19.093015 752 consensus_queue.cc:237] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.093261 900 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:00:19.094861 32428 leader_election.cc:341] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.087273 357 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.095795 32615 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
W20260806 04:00:19.096659 32553 leader_election.cc:341] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.096699 32427 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.097606 900 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:19.098074 900 raft_consensus.cc:686] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:19.098897 900 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.119308 351 tablet_bootstrap.cc:654] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.121042 583 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.121212 764 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.121549 583 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.121606 764 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.123942 583 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:19.124511 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.129443 32749 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:19.127269 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:19.130259 32748 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.130798 357 ts_tablet_manager.cc:1428] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.047s user 0.002s sys 0.006s
I20260806 04:00:19.133076 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.142019 357 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:19.143702 32748 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:19.127588 352 tablet_bootstrap.cc:654] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.149833 32326 catalog_manager.cc:5495] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 3, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:19.128813 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.151928 32615 raft_consensus.cc:2379] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:19.152838 764 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:19.153064 32682 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:19.153422 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.154069 32478 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:19.154827 583 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.155014 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:19.157008 32682 leader_election.cc:341] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.161300 32683 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:19.161717 357 tablet_bootstrap.cc:654] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.162264 583 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:19.162662 583 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.162947 583 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.167573 32321 catalog_manager.cc:5495] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:19.169467 583 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.173012 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:00:19.174121 32682 leader_election.cc:341] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.171380 583 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:19.175912 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:19.176792 752 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.178355 752 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.180224 752 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.182646 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.178350 32478 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.183286 32477 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:19.184489 32553 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.185354 752 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:19.185750 752 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.186120 752 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.188609 900 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.189069 900 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.191808 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.192734 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:19.193002 32427 leader_election.cc:341] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:19.194029 32428 leader_election.cc:341] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.194406 32428 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:19.195125 454 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.195691 900 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.196878 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.199036 752 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:19.200472 32553 leader_election.cc:341] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.202126 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:19.202719 32477 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.203426 352 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:19.203799 352 ts_tablet_manager.cc:1397] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.120s user 0.028s sys 0.021s
I20260806 04:00:19.203083 32324 catalog_manager.cc:5495] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:19.204814 752 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.206009 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:19.206264 352 raft_consensus.cc:348] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.206851 352 raft_consensus.cc:374] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.207101 352 raft_consensus.cc:729] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
W20260806 04:00:19.207137 32553 leader_election.cc:341] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.207653 351 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:19.208127 351 ts_tablet_manager.cc:1397] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.126s user 0.013s sys 0.000s
I20260806 04:00:19.207762 352 consensus_queue.cc:260] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.205364 32324 catalog_manager.cc:5495] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:19.209339 352 ts_tablet_manager.cc:1428] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260806 04:00:19.210090 352 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:19.210325 351 raft_consensus.cc:348] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.210961 351 raft_consensus.cc:374] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.211202 351 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.214763 351 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.214005 32324 catalog_manager.cc:5495] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:19.217767 32478 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:19.218338 351 ts_tablet_manager.cc:1428] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260806 04:00:19.219362 351 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:19.221334 352 tablet_bootstrap.cc:654] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.222913 32683 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:19.223709 920 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:19.223917 32477 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:19.224236 920 raft_consensus.cc:686] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:19.225144 32553 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:19.225181 920 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.229221 357 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:19.229169 752 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:19.229775 357 ts_tablet_manager.cc:1397] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.009s
I20260806 04:00:19.231086 752 raft_consensus.cc:686] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:19.231978 752 consensus_queue.cc:237] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.232293 357 raft_consensus.cc:348] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.233070 357 raft_consensus.cc:374] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.233347 357 raft_consensus.cc:729] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.236543 357 consensus_queue.cc:260] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.248436 357 ts_tablet_manager.cc:1428] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
I20260806 04:00:19.249523 357 tablet_bootstrap.cc:492] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:19.253695 32324 catalog_manager.cc:5495] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:19.256201 351 tablet_bootstrap.cc:654] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.257503 357 tablet_bootstrap.cc:654] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.261675 352 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:19.262182 352 ts_tablet_manager.cc:1397] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.015s
I20260806 04:00:19.265008 352 raft_consensus.cc:348] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.271505 352 raft_consensus.cc:374] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.271788 352 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.272399 352 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.274484 352 ts_tablet_manager.cc:1428] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 04:00:19.274662 32321 catalog_manager.cc:5495] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:19.278009 351 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:19.278668 351 ts_tablet_manager.cc:1397] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.000s
I20260806 04:00:19.281196 351 raft_consensus.cc:348] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.281740 351 raft_consensus.cc:374] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.281997 351 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.282830 351 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.283118 900 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.283666 900 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.285712 900 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.286058 352 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Bootstrap starting.
I20260806 04:00:19.288811 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:19.289798 32427 leader_election.cc:341] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.291579 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.292649 351 ts_tablet_manager.cc:1428] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
W20260806 04:00:19.293504 32428 leader_election.cc:341] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.293767 352 tablet_bootstrap.cc:654] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.294678 351 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:19.293915 32428 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:19.294883 357 tablet_bootstrap.cc:492] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:19.295311 900 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.295997 357 ts_tablet_manager.cc:1397] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.016s
I20260806 04:00:19.301625 351 tablet_bootstrap.cc:654] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.302304 357 raft_consensus.cc:348] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.302994 357 raft_consensus.cc:374] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.303234 357 raft_consensus.cc:729] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.303913 357 consensus_queue.cc:260] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.306031 357 ts_tablet_manager.cc:1428] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260806 04:00:19.306927 900 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.307397 900 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.309438 900 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.310987 357 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:19.312954 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:19.314601 32427 leader_election.cc:341] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.316898 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:19.318573 32428 leader_election.cc:341] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.319052 32428 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:19.319835 900 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.321568 357 tablet_bootstrap.cc:654] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.323696 351 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:19.323769 352 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: No bootstrap required, opened a new log
I20260806 04:00:19.324229 351 ts_tablet_manager.cc:1397] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.001s
I20260806 04:00:19.324338 352 ts_tablet_manager.cc:1397] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.000s
I20260806 04:00:19.333634 351 raft_consensus.cc:348] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.334400 351 raft_consensus.cc:374] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.334722 351 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.335425 351 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.336356 352 raft_consensus.cc:348] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.337081 352 raft_consensus.cc:374] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.337347 352 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.338083 352 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.339752 351 ts_tablet_manager.cc:1428] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.015s user 0.004s sys 0.002s
I20260806 04:00:19.340452 352 ts_tablet_manager.cc:1428] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20260806 04:00:19.341590 357 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:19.342013 357 ts_tablet_manager.cc:1397] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.000s
I20260806 04:00:19.343420 351 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:19.349952 357 raft_consensus.cc:348] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.350725 357 raft_consensus.cc:374] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.350991 357 raft_consensus.cc:729] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.351653 357 consensus_queue.cc:260] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.353912 357 ts_tablet_manager.cc:1428] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260806 04:00:19.356683 351 tablet_bootstrap.cc:654] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.357578 900 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:19.362547 357 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:19.364871 454 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.365621 454 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.369200 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:00:19.370718 32427 leader_election.cc:341] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.371783 454 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.373606 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.374557 357 tablet_bootstrap.cc:654] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
W20260806 04:00:19.382021 32428 leader_election.cc:341] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.382611 32428 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:19.383838 454 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.385440 752 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.385967 752 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.389696 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.391048 357 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:19.391517 357 ts_tablet_manager.cc:1397] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.008s
I20260806 04:00:19.394057 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.391508 32477 raft_consensus.cc:2379] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 1: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:00:19.396900 32553 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:19.397591 795 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.399205 454 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:19.400491 357 raft_consensus.cc:348] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.401196 357 raft_consensus.cc:374] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.401511 357 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.402418 357 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.404810 357 ts_tablet_manager.cc:1428] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260806 04:00:19.406450 357 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:19.406987 752 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.410189 795 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.414332 357 tablet_bootstrap.cc:654] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.423401 357 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:19.423835 357 ts_tablet_manager.cc:1397] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20260806 04:00:19.425012 795 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.425710 795 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.426900 32748 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.427263 357 raft_consensus.cc:348] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.427922 357 raft_consensus.cc:374] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.428191 357 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.428753 357 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.433955 795 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.445889 357 ts_tablet_manager.cc:1428] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260806 04:00:19.444878 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.445787 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.463236 795 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.477486 795 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.480801 32745 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:19.486272 357 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:19.488063 32553 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:19.488914 590 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:19.489303 590 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.489636 590 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.489744 795 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.493757 351 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:19.494231 351 ts_tablet_manager.cc:1397] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.151s user 0.024s sys 0.020s
I20260806 04:00:19.498513 732 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:19.490064 32477 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:19.498404 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.500309 32478 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:19.502406 32553 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.505894 795 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:19.506829 795 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.507429 795 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.500308 32608 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
I20260806 04:00:19.509279 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.506175 351 raft_consensus.cc:348] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.524374 351 raft_consensus.cc:374] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.524732 351 raft_consensus.cc:729] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.515972 590 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.519071 454 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:19.528317 675 consensus_queue.cc:1046] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 04:00:19.530085 32553 leader_election.cc:341] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.530934 583 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.531351 675 raft_consensus.cc:3032] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:00:19.531853 675 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:19.531389 583 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.532933 675 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.534171 675 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:19.534471 583 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:19.534759 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.535689 32478 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:19.537500 32683 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:00:19.525249 32428 consensus_peers.cc:487] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Couldn't send request to peer f7c202b562ce482db283463d7f02e136. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:19.539207 764 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:19.525427 351 consensus_queue.cc:260] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.539656 764 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.540194 764 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.552853 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.553906 32608 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:19.564904 351 ts_tablet_manager.cc:1428] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.070s user 0.007s sys 0.011s
I20260806 04:00:19.566231 351 tablet_bootstrap.cc:492] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:19.570719 357 tablet_bootstrap.cc:654] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.573645 454 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:19.579208 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:19.589637 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:19.590428 32743 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.594132 32610 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:00:19.586413 32478 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.606694 351 tablet_bootstrap.cc:654] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.580571 590 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.662593 32743 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:19.666339 32478 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:19.669304 764 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.669719 32553 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:19.670591 590 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:19.671221 590 raft_consensus.cc:686] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:19.671509 764 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:19.672529 590 consensus_queue.cc:237] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.674787 718 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.675300 718 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.677835 718 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:19.693321 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:19.694015 32475 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:19.696571 32683 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:00:19.699782 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 04:00:19.704725 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:19.692456 718 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260806 04:00:19.708485 920 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:19.729741 920 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.730291 920 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.723986 32478 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.748558 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:19.749004 351 tablet_bootstrap.cc:492] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:19.749239 32609 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.749459 351 ts_tablet_manager.cc:1397] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.183s user 0.016s sys 0.003s
I20260806 04:00:19.753263 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.755692 357 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:19.756217 357 ts_tablet_manager.cc:1397] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.270s user 0.017s sys 0.040s
I20260806 04:00:19.755964 351 raft_consensus.cc:348] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.756659 351 raft_consensus.cc:374] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.742136 900 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.758317 351 raft_consensus.cc:729] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.758958 351 consensus_queue.cc:260] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.760360 357 raft_consensus.cc:348] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.765540 357 raft_consensus.cc:374] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:19.765842 357 raft_consensus.cc:729] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:19.766507 357 consensus_queue.cc:260] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.771191 675 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.763684 900 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.771761 675 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.773126 900 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.773954 675 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.742502 752 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260806 04:00:19.760957 948 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260806 04:00:19.775149 899 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:19.771888 795 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.787969 583 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:19.791226 32748 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.792891 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:19.793447 32475 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.796463 351 ts_tablet_manager.cc:1428] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.047s user 0.005s sys 0.003s
I20260806 04:00:19.782486 920 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.805887 795 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:19.806732 357 ts_tablet_manager.cc:1428] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20260806 04:00:19.807629 357 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:19.807273 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.808154 32608 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:19.810068 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:19.786924 32478 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:19.834659 32615 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:19.857538 351 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:19.824452 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:19.876892 32475 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:19.826680 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:19.895799 947 log.cc:924] Time spent T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.075s user 0.006s sys 0.000s
I20260806 04:00:19.899164 32749 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:19.900815 764 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:19.902486 920 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:00:19.910476 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 87574us
I20260806 04:00:19.913192 32427 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.915773 32553 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:19.926214 32683 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:19.926945 583 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:19.927402 583 raft_consensus.cc:686] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:00:19.923806 32428 leader_election.cc:341] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:19.930176 32427 leader_election.cc:341] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.928251 583 consensus_queue.cc:237] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.933332 732 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 04:00:19.936954 959 log.cc:924] Time spent T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20260806 04:00:19.937044 732 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:19.937218 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:19.937620 732 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:00:19.938786 32553 consensus_peers.cc:487] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 -> Peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Couldn't send request to peer f7c202b562ce482db283463d7f02e136. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260806 04:00:19.939607 32553 leader_election.cc:341] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:19.909582 764 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:19.929076 958 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260806 04:00:19.956489 32427 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:19.948457 32609 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:19.969161 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:19.969877 32605 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:19.962885 948 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:19.985493 966 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:19.987349 32605 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:19.995129 357 tablet_bootstrap.cc:654] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:19.996258 32473 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 04:00:19.996795 32473 pending_rounds.cc:76] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 1
I20260806 04:00:20.004375 583 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260806 04:00:19.997207 32473 pending_rounds.cc:97] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 04:00:20.009684 966 raft_consensus.cc:686] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:20.028611 32473 raft_consensus.cc:2866] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:00:20.038738 764 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:20.046082 351 tablet_bootstrap.cc:654] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:20.055821 966 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.094492 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:20.095172 32475 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:00:20.104305 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 93964.8us
I20260806 04:00:20.106124 32682 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:20.110565 583 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:20.111028 583 raft_consensus.cc:686] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:20.111765 583 consensus_queue.cc:237] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.125929 32475 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:20.123387 732 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.187770 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:20.186419 718 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260806 04:00:20.197750 32321 catalog_manager.cc:5495] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
W20260806 04:00:20.221683 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 162233us
W20260806 04:00:20.236063 995 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.094s user 0.005s sys 0.012s
W20260806 04:00:20.254462 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 191380us
I20260806 04:00:20.284809 732 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:00:20.306434 32682 leader_election.cc:341] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:20.325657 752 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:20.341594 351 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:20.342087 351 ts_tablet_manager.cc:1397] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.485s user 0.016s sys 0.008s
I20260806 04:00:20.347065 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:20.348011 32469 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:20.368353 351 raft_consensus.cc:348] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.369155 351 raft_consensus.cc:374] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:20.369349 357 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:20.369421 351 raft_consensus.cc:729] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:20.369886 357 ts_tablet_manager.cc:1397] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.562s user 0.028s sys 0.034s
I20260806 04:00:20.370096 351 consensus_queue.cc:260] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.372208 357 raft_consensus.cc:348] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.373011 357 raft_consensus.cc:374] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:20.372519 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:20.373292 357 raft_consensus.cc:729] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:20.373492 32749 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:20.373942 357 consensus_queue.cc:260] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.375751 351 ts_tablet_manager.cc:1428] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260806 04:00:20.376693 351 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:20.382874 357 ts_tablet_manager.cc:1428] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260806 04:00:20.387409 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:20.388409 32609 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:20.390920 32749 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:20.401973 32609 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:20.404559 32428 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:20.406287 32683 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:20.411202 991 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:20.413528 991 raft_consensus.cc:686] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:20.414930 351 tablet_bootstrap.cc:654] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:20.414678 991 consensus_queue.cc:237] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.422945 357 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:20.422364 920 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:20.426674 920 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:20.427019 920 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:20.473910 967 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260806 04:00:20.450513 920 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:20.512794 1007 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260806 04:00:20.555070 920 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:20.556241 357 tablet_bootstrap.cc:654] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
W20260806 04:00:20.566920 1012 log.cc:924] Time spent T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260806 04:00:20.575946 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:20.576502 32475 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:20.589920 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:20.593400 32475 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:20.611768 583 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:20.599475 962 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260806 04:00:20.638337 351 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:20.638845 351 ts_tablet_manager.cc:1397] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.262s user 0.013s sys 0.033s
W20260806 04:00:20.640431 32682 leader_election.cc:341] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:20.647063 795 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 04:00:20.658380 1015 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.092s user 0.000s sys 0.002s
I20260806 04:00:20.676261 718 consensus_queue.cc:1035] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 04:00:20.680714 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 69656us
I20260806 04:00:20.691943 32683 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
W20260806 04:00:20.708760 1017 log.cc:924] Time spent T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.069s user 0.004s sys 0.000s
I20260806 04:00:20.719569 965 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:20.720393 965 raft_consensus.cc:686] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:20.736038 351 raft_consensus.cc:348] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.736902 351 raft_consensus.cc:374] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:20.737152 351 raft_consensus.cc:729] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:20.737720 351 consensus_queue.cc:260] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.742792 351 ts_tablet_manager.cc:1428] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.104s user 0.006s sys 0.002s
I20260806 04:00:20.745042 351 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:20.743188 965 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.776656 998 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:00:20.780225 732 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:20.788915 998 consensus_queue.cc:1035] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260806 04:00:20.790308 969 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:20.790881 969 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:20.809430 32553 consensus_peers.cc:487] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 -> Peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Couldn't send request to peer f7c202b562ce482db283463d7f02e136. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:20.788481 732 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:20.865115 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:20.865813 32478 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:20.868784 732 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:20.870124 764 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:20.870707 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:20.870617 764 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:20.872320 32427 leader_election.cc:341] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:20.872931 764 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:20.870646 969 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:20.874261 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:20.875135 32608 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:20.879746 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:20.880590 764 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:20.881026 764 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:20.881556 764 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:20.885049 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:20.885507 32553 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:20.905720 1028 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:20.906294 1028 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:20.906864 1028 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:20.914765 351 tablet_bootstrap.cc:654] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
W20260806 04:00:20.938346 1038 log.cc:924] Time spent T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260806 04:00:20.938848 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:20.950019 32553 leader_election.cc:341] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
W20260806 04:00:20.950773 32428 leader_election.cc:341] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:20.956245 32428 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:00:20.960382 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:20.961371 32467 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 0.
I20260806 04:00:20.966295 32609 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 04:00:20.971473 732 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:20.976238 732 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:20.991494 357 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:20.992012 357 ts_tablet_manager.cc:1397] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.569s user 0.024s sys 0.030s
I20260806 04:00:21.002460 357 raft_consensus.cc:348] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.003193 357 raft_consensus.cc:374] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.003477 357 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.003513 32324 catalog_manager.cc:5495] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:21.004154 357 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.005511 32321 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39766 (request call id 105) took 1073 ms. Trace:
I20260806 04:00:21.009300 357 ts_tablet_manager.cc:1428] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260806 04:00:21.010377 357 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
W20260806 04:00:21.019006 32400 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260806 04:00:21.029752 1035 log.cc:924] Time spent T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260806 04:00:21.031152 32741 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 04:00:21.005872 32321 rpcz_store.cc:276] 0806 04:00:19.940408 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:19.941736 (+ 1328us) service_pool.cc:227] Handling call
0806 04:00:20.996538 (+1054802us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:00:20.364768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:00:20.365292 (+ 524us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:00:20.365347 (+ 55us) write_op.cc:435] Acquired schema lock
0806 04:00:20.365366 (+ 19us) tablet.cc:660] Decoding operations
0806 04:00:20.365700 (+ 334us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:00:20.365771 (+ 71us) write_op.cc:641] Partition lock acquired for write op
0806 04:00:20.365799 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:00:20.365972 (+ 173us) tablet.cc:699] Row locks acquired
0806 04:00:20.365988 (+ 16us) write_op.cc:260] PREPARE: finished
0806 04:00:20.366116 (+ 128us) write_op.cc:270] Start()
0806 04:00:20.366217 (+ 101us) write_op.cc:276] Timestamp: P: 1785988820366081 usec, L: 0
0806 04:00:20.366236 (+ 19us) op_driver.cc:331] REPLICATION: starting
0806 04:00:20.366904 (+ 668us) log.cc:842] Serialized 543 byte log entry
0806 04:00:20.613302 (+246398us) op_driver.cc:444] REPLICATION: finished
0806 04:00:20.866701 (+253399us) write_op.cc:301] APPLY: starting
0806 04:00:20.891744 (+ 25043us) tablet.cc:1365] starting BulkCheckPresence
0806 04:00:20.915706 (+ 23962us) tablet.cc:1368] finished BulkCheckPresence
0806 04:00:20.977697 (+ 61991us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:00:20.978054 (+ 357us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:00:20.978431 (+ 377us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:00:20.978475 (+ 44us) write_op.cc:312] APPLY: finished
0806 04:00:20.978661 (+ 186us) log.cc:842] Serialized 22 byte log entry
0806 04:00:20.980092 (+ 1431us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:00:20.980212 (+ 120us) write_op.cc:454] Released schema lock
0806 04:00:20.980262 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0806 04:00:20.980593 (+ 331us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:00:20.980633 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":223485,"apply.run_cpu_time_us":7597,"apply.run_wall_time_us":139924,"num_ops":1,"prepare.queue_time_us":29956,"prepare.run_cpu_time_us":2904,"prepare.run_wall_time_us":23513,"raft.queue_time_us":98871,"raft.run_cpu_time_us":39779,"raft.run_wall_time_us":252584,"replication_time_us":246905,"thread_start_us":380044,"threads_started":4,"wal-append.queue_time_us":18073}]]}
I20260806 04:00:21.056970 732 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260806 04:00:21.086290 764 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.092954 764 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:21.094448 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:21.097788 32608 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.098282 989 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:21.098898 989 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.103900 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:21.104959 32466 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.112416 351 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:21.112900 351 ts_tablet_manager.cc:1397] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.368s user 0.038s sys 0.042s
I20260806 04:00:21.113201 989 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:21.121935 357 tablet_bootstrap.cc:654] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.122781 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:21.130470 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:21.133467 1028 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.136268 32682 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:21.141357 989 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:21.154196 351 raft_consensus.cc:348] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.154838 351 raft_consensus.cc:374] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.155133 351 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.156615 351 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.151338 1028 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.152805 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:21.155725 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:21.163439 351 ts_tablet_manager.cc:1428] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20260806 04:00:21.166105 351 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:21.161881 32467 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.172591 32608 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
W20260806 04:00:21.170090 32553 leader_election.cc:341] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:21.181973 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:21.183005 989 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:21.188337 32326 catalog_manager.cc:5495] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 2, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:21.186546 989 raft_consensus.cc:686] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:21.206323 989 consensus_queue.cc:237] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.209833 32467 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:21.211465 32466 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:00:21.212029 32553 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:21.212776 1033 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:21.213264 1033 raft_consensus.cc:686] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
W20260806 04:00:21.214118 32682 consensus_peers.cc:487] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 -> Peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): Couldn't send request to peer dd7344d85a154539b7f8c484f2a76772. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:21.214226 1033 consensus_queue.cc:237] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.220162 351 tablet_bootstrap.cc:654] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.230849 351 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:21.231374 351 ts_tablet_manager.cc:1397] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.001s
I20260806 04:00:21.234148 351 raft_consensus.cc:348] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.234988 351 raft_consensus.cc:374] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.235311 351 raft_consensus.cc:729] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.236042 351 consensus_queue.cc:260] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.238674 351 ts_tablet_manager.cc:1428] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 04:00:21.239595 351 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:21.243302 357 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:21.243760 357 ts_tablet_manager.cc:1397] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.234s user 0.009s sys 0.008s
I20260806 04:00:21.245034 351 tablet_bootstrap.cc:654] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.246119 357 raft_consensus.cc:348] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.246863 357 raft_consensus.cc:374] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.247093 357 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.247709 357 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.250031 357 ts_tablet_manager.cc:1428] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:00:21.250946 357 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:21.255146 32324 catalog_manager.cc:5495] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:00:21.258090 32324 catalog_manager.cc:5495] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:21.261746 357 tablet_bootstrap.cc:654] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.274173 32741 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.289381 357 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:21.289011 764 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:21.289898 357 ts_tablet_manager.cc:1397] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.017s
I20260806 04:00:21.295872 357 raft_consensus.cc:348] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.296641 357 raft_consensus.cc:374] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.296887 357 raft_consensus.cc:729] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.297578 357 consensus_queue.cc:260] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.300122 357 ts_tablet_manager.cc:1428] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260806 04:00:21.301178 357 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:21.301942 351 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:21.302397 351 ts_tablet_manager.cc:1397] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.011s
I20260806 04:00:21.305734 351 raft_consensus.cc:348] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.306433 351 raft_consensus.cc:374] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.306708 351 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.307507 351 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.308827 357 tablet_bootstrap.cc:654] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.318301 1028 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:21.319036 1028 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.322932 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:21.323701 32466 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:21.325075 32553 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:21.325254 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:21.326028 32745 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:21.327054 966 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:21.327458 966 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:21.327922 966 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.330792 1028 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.339957 732 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:21.340703 732 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.343736 732 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.345098 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:21.351683 32428 leader_election.cc:341] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:21.341130 966 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.347492 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:21.357241 32608 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:21.357561 351 ts_tablet_manager.cc:1428] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.055s user 0.009s sys 0.001s
I20260806 04:00:21.349378 1033 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:21.358676 351 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:21.358716 1033 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.361469 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:21.362475 32745 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.363080 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:21.363889 32469 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.367467 1033 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.345393 32321 catalog_manager.cc:5495] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:21.368525 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:21.372641 32321 catalog_manager.cc:5495] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:21.372480 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:21.368861 32427 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:21.373279 32467 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 0.
I20260806 04:00:21.376606 732 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:21.376979 732 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:00:21.376565 32553 leader_election.cc:341] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:21.377276 732 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.379007 32553 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:21.380033 976 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:21.380530 976 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:21.380961 976 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.382112 351 tablet_bootstrap.cc:654] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.385160 732 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.384589 966 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.392400 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:00:21.396853 32428 leader_election.cc:341] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:21.399660 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:21.400434 32614 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.407101 732 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.412678 32745 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:21.413936 32553 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:21.414629 1033 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:21.415324 1033 raft_consensus.cc:686] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:21.415644 32469 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:21.416687 1033 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.430747 32324 catalog_manager.cc:5495] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:21.437851 357 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:21.438383 357 ts_tablet_manager.cc:1397] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.137s user 0.021s sys 0.035s
I20260806 04:00:21.443631 357 raft_consensus.cc:348] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.444366 357 raft_consensus.cc:374] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.448376 357 raft_consensus.cc:729] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.449995 357 consensus_queue.cc:260] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.454613 357 ts_tablet_manager.cc:1428] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 04:00:21.447608 32614 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:21.456619 976 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.458738 32427 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:21.460229 976 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.463745 357 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:21.460777 1032 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:21.469022 1032 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:21.467377 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:21.470315 32745 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.470181 1032 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.467496 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:21.473733 32469 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.474423 351 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:21.474913 351 ts_tablet_manager.cc:1397] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.116s user 0.009s sys 0.005s
I20260806 04:00:21.481899 351 raft_consensus.cc:348] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.482694 351 raft_consensus.cc:374] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.482998 351 raft_consensus.cc:729] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.483646 351 consensus_queue.cc:260] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.485087 32324 catalog_manager.cc:5495] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:21.488799 351 ts_tablet_manager.cc:1428] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.014s user 0.009s sys 0.001s
I20260806 04:00:21.490290 357 tablet_bootstrap.cc:654] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.492456 32745 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:21.500602 351 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: Bootstrap starting.
I20260806 04:00:21.502149 32553 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:21.502955 976 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:21.503729 976 raft_consensus.cc:686] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:21.505620 732 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:21.506063 732 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.505095 976 consensus_queue.cc:237] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.514595 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:21.515419 32614 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 0.
I20260806 04:00:21.516692 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:21.517558 993 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:00:21.517990 993 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:00:21.513628 732 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.518417 993 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.524338 32326 catalog_manager.cc:5495] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:21.534907 32469 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:00:21.548796 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:00:21.550640 32428 leader_election.cc:341] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:21.561450 998 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:21.562393 998 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.566628 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:21.569026 32466 raft_consensus.cc:2379] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:00:21.569633 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:21.568182 32326 catalog_manager.cc:5495] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:21.567334 993 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.572769 32682 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:21.573462 351 tablet_bootstrap.cc:654] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.573804 920 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.575187 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:21.575769 32609 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.577916 993 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:21.570843 998 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:21.582330 357 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:21.582849 357 ts_tablet_manager.cc:1397] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.119s user 0.011s sys 0.005s
I20260806 04:00:21.590742 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:21.590549 357 raft_consensus.cc:348] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:00:21.591957 32428 leader_election.cc:341] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): Illegal state: must be running to vote when last-logged opid is not known
I20260806 04:00:21.592275 357 raft_consensus.cc:374] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.592567 357 raft_consensus.cc:729] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.593171 357 consensus_queue.cc:260] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.594806 357 ts_tablet_manager.cc:1428] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260806 04:00:21.595824 357 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:21.602264 351 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: No bootstrap required, opened a new log
I20260806 04:00:21.602701 351 ts_tablet_manager.cc:1397] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: Time spent bootstrapping tablet: real 0.102s user 0.019s sys 0.013s
I20260806 04:00:21.600337 920 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:21.598788 32609 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:00:21.608893 351 raft_consensus.cc:348] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.609568 351 raft_consensus.cc:374] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.609623 920 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:21.610090 351 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.609915 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:00:21.611003 993 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:00:21.611828 351 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.612408 993 raft_consensus.cc:686] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:21.614563 993 consensus_queue.cc:237] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.629901 357 tablet_bootstrap.cc:654] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.634826 32324 catalog_manager.cc:5495] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:21.637694 32609 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.647179 998 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:21.662109 976 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:21.673372 351 ts_tablet_manager.cc:1428] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: Time spent starting tablet: real 0.070s user 0.008s sys 0.000s
I20260806 04:00:21.684063 976 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 04:00:21.697679 357 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:21.698184 357 ts_tablet_manager.cc:1397] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.103s user 0.014s sys 0.005s
I20260806 04:00:21.704192 357 raft_consensus.cc:348] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.704886 357 raft_consensus.cc:374] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.705145 357 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.705734 357 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.707772 357 ts_tablet_manager.cc:1428] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260806 04:00:21.708673 357 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:21.726516 32741 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.733477 357 tablet_bootstrap.cc:654] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.749960 357 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:21.750475 357 ts_tablet_manager.cc:1397] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.001s
I20260806 04:00:21.761505 357 raft_consensus.cc:348] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.762224 357 raft_consensus.cc:374] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.762498 357 raft_consensus.cc:729] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.763067 357 consensus_queue.cc:260] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.767184 357 ts_tablet_manager.cc:1428] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260806 04:00:21.768780 357 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: Bootstrap starting.
I20260806 04:00:21.778031 357 tablet_bootstrap.cc:654] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: Neither blocks nor log segments found. Creating new log.
I20260806 04:00:21.793157 357 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: No bootstrap required, opened a new log
I20260806 04:00:21.793587 357 ts_tablet_manager.cc:1397] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.000s
I20260806 04:00:21.796450 357 raft_consensus.cc:348] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.797132 357 raft_consensus.cc:374] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:00:21.797386 357 raft_consensus.cc:729] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Initialized, Role: FOLLOWER
I20260806 04:00:21.797983 357 consensus_queue.cc:260] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.799986 357 ts_tablet_manager.cc:1428] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 04:00:21.844621 764 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:21.845136 764 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.847138 764 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:21.847747 764 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:00:21.848006 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:21.848253 764 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:21.848896 32614 raft_consensus.cc:2379] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 1: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:00:21.849387 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:21.850273 764 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:21.850867 32683 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:21.851027 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:21.851329 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:00:21.851583 764 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:21.852613 32682 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:21.853251 998 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:21.857565 764 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:00:21.928650 1028 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:21.938081 966 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:21.945081 993 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:21.966143 948 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:22.007970 966 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:22.024847 976 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:22.025646 32745 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:22.051225 32739 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:00:22.067834 993 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 04:00:22.109834 993 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:00:22.968217 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:00:23.006403 1116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:23.006433 1119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:00:23.007432 1124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:00:23.008701 30646 server_base.cc:1034] running on GCE node
I20260806 04:00:23.010290 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 04:00:23.010603 30646 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 04:00:23.010793 30646 hybrid_clock.cc:648] HybridClock initialized: now 1785988823010778 us; error 0 us; skew 500 ppm
I20260806 04:00:23.011520 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 04:00:23.015286 30646 webserver.cc:460] Webserver started at http://127.29.237.132:37655/ using document root <none> and password file <none>
I20260806 04:00:23.017077 30646 fs_manager.cc:359] Metadata directory not provided
I20260806 04:00:23.017345 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:00:23.017771 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:00:23.019419 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-3-root/instance:
uuid: "d4a238f055a44f459c62c224daa4d541"
format_stamp: "Formatted at 2026-08-06 04:00:23 on dist-test-slave-lgr5"
I20260806 04:00:23.024663 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 04:00:23.028662 1132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:23.029909 30646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 04:00:23.030206 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-3-root
uuid: "d4a238f055a44f459c62c224daa4d541"
format_stamp: "Formatted at 2026-08-06 04:00:23 on dist-test-slave-lgr5"
I20260806 04:00:23.030571 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:00:23.057745 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:00:23.059466 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:00:23.062052 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:00:23.065366 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:00:23.065567 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:23.065857 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:00:23.066015 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:00:23.151198 30646 rpc_server.cc:307] RPC server started. Bound to: 127.29.237.132:36181
I20260806 04:00:23.151285 1231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.237.132:36181 every 8 connection(s)
I20260806 04:00:23.180112 30646 auto_leader_rebalancer-test.cc:413] add a tserver: d4a238f055a44f459c62c224daa4d541
I20260806 04:00:23.189786 1233 heartbeater.cc:347] Connected to a master server at 127.29.237.190:40089
I20260806 04:00:23.190290 1233 heartbeater.cc:464] Registering TS with master...
I20260806 04:00:23.191090 1233 heartbeater.cc:511] Master 127.29.237.190:40089 requested a full tablet report, sending...
I20260806 04:00:23.193061 32324 ts_manager.cc:194] Registered new tserver with Master: d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:23.194556 32324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36880
I20260806 04:00:24.197486 1233 heartbeater.cc:503] Master 127.29.237.190:40089 was elected leader, sending a full tablet report...
I20260806 04:00:24.253444 32467 consensus_queue.cc:237] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:24.258563 32739 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:24.259828 948 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:24.261176 32614 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:24.264305 732 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:24.270437 732 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.274027 32745 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.276198 32608 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
W20260806 04:00:24.284575 32427 consensus_peers.cc:487] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c9b09ac79f84fb797c2a263c612eee5. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:24.290791 32324 catalog_manager.cc:5495] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:24.336848 32745 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:24.363651 32469 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:24.370713 1258 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:24.372824 32614 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:24.374512 1258 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:24.393018 32469 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.391525 764 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
W20260806 04:00:24.399224 32682 consensus_peers.cc:487] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ec5d99c8efa405b84db49741c2f8933. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:24.418218 32321 catalog_manager.cc:5495] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:24.425452 32614 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.459607 32614 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:24.469165 32745 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:24.472909 32467 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:24.472935 1033 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 04:00:24.477164 976 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 04:00:24.500568 32551 consensus_peers.cc:487] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bf6a30af60042f9b0a84eb0fdb8db19. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:24.508642 1280 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.510478 32467 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.520516 32745 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.527815 32469 consensus_queue.cc:237] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:24.535983 32326 catalog_manager.cc:5495] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:24.584847 32614 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:24.585970 32739 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:24.586992 32427 consensus_peers.cc:487] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d20f5ce3f2ac40379306e3d630f63eda. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:24.589859 732 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:24.594424 732 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:24.608299 1287 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.611500 32745 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.616266 32614 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.634260 32326 catalog_manager.cc:5495] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:24.698640 1296 ts_tablet_manager.cc:927] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:24.711844 1296 tablet_copy_client.cc:316] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:24.743782 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 3c9b09ac79f84fb797c2a263c612eee5 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:24.744756 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 3c9b09ac79f84fb797c2a263c612eee5 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-3c9b09ac79f84fb797c2a263c612eee5
I20260806 04:00:24.755394 32506 tablet_copy_source_session.cc:205] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:24.760411 1296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9b09ac79f84fb797c2a263c612eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:24.781805 1296 tablet_copy_client.cc:791] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:24.782697 1296 tablet_copy_client.cc:655] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:24.787681 1296 tablet_copy_client.cc:531] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:24.791281 1299 ts_tablet_manager.cc:927] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:24.800355 1296 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:24.803615 1299 tablet_copy_client.cc:316] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:24.833665 300 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet 7ec5d99c8efa405b84db49741c2f8933 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:24.834622 300 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet 7ec5d99c8efa405b84db49741c2f8933 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-7ec5d99c8efa405b84db49741c2f8933
I20260806 04:00:24.849401 300 tablet_copy_source_session.cc:205] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:24.854895 1299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec5d99c8efa405b84db49741c2f8933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:24.857685 1296 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:24.858937 1296 tablet_bootstrap.cc:492] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:24.859742 1296 ts_tablet_manager.cc:1397] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.060s user 0.017s sys 0.024s
I20260806 04:00:24.868551 1296 raft_consensus.cc:348] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:24.869800 1296 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:24.870779 1296 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:24.874807 1303 ts_tablet_manager.cc:927] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:24.889551 1299 tablet_copy_client.cc:791] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:24.890352 1299 tablet_copy_client.cc:655] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:24.908335 1303 tablet_copy_client.cc:316] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:24.908565 1296 ts_tablet_manager.cc:1428] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.048s user 0.011s sys 0.008s
I20260806 04:00:24.913522 32506 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-3c9b09ac79f84fb797c2a263c612eee5 received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:24.914018 32506 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-3c9b09ac79f84fb797c2a263c612eee5 on tablet 3c9b09ac79f84fb797c2a263c612eee5 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:24.915402 1299 tablet_copy_client.cc:531] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:24.926687 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 0bf6a30af60042f9b0a84eb0fdb8db19 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:24.927337 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 0bf6a30af60042f9b0a84eb0fdb8db19 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-0bf6a30af60042f9b0a84eb0fdb8db19
I20260806 04:00:24.928607 1299 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:24.935829 32631 tablet_copy_source_session.cc:205] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:24.940505 1303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6a30af60042f9b0a84eb0fdb8db19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:24.962481 1303 tablet_copy_client.cc:791] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:24.963280 1303 tablet_copy_client.cc:655] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:24.967947 1303 tablet_copy_client.cc:531] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:24.969987 1299 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:24.971062 1299 tablet_bootstrap.cc:492] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:24.971958 1299 ts_tablet_manager.cc:1397] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.044s user 0.028s sys 0.005s
I20260806 04:00:24.976712 1299 raft_consensus.cc:348] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:24.977748 1299 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:24.979282 1296 ts_tablet_manager.cc:927] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:24.978834 1299 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:24.981626 1303 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:24.985667 1299 ts_tablet_manager.cc:1428] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.013s user 0.008s sys 0.002s
I20260806 04:00:24.990358 300 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-7ec5d99c8efa405b84db49741c2f8933 received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:24.990828 300 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-7ec5d99c8efa405b84db49741c2f8933 on tablet 7ec5d99c8efa405b84db49741c2f8933 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:25.005545 1296 tablet_copy_client.cc:316] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:25.008780 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet d20f5ce3f2ac40379306e3d630f63eda from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:25.009423 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet d20f5ce3f2ac40379306e3d630f63eda from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-d20f5ce3f2ac40379306e3d630f63eda
I20260806 04:00:25.022526 32506 tablet_copy_source_session.cc:205] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:25.023088 1303 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:25.024336 1303 tablet_bootstrap.cc:492] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:25.025244 1303 ts_tablet_manager.cc:1397] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.044s user 0.018s sys 0.019s
I20260806 04:00:25.026844 1296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20f5ce3f2ac40379306e3d630f63eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:25.031349 1303 raft_consensus.cc:348] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:25.032555 1303 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:25.033509 1303 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:25.037012 1303 ts_tablet_manager.cc:1428] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260806 04:00:25.039242 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-0bf6a30af60042f9b0a84eb0fdb8db19 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:25.039808 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-0bf6a30af60042f9b0a84eb0fdb8db19 on tablet 0bf6a30af60042f9b0a84eb0fdb8db19 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:25.050215 1296 tablet_copy_client.cc:791] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:25.050951 1296 tablet_copy_client.cc:655] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:25.055269 1296 tablet_copy_client.cc:531] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:25.070098 1296 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:25.102250 1296 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:25.103045 1296 tablet_bootstrap.cc:492] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:25.103694 1296 ts_tablet_manager.cc:1397] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.012s
I20260806 04:00:25.107857 1296 raft_consensus.cc:348] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:25.109141 1296 raft_consensus.cc:729] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:25.110049 1296 consensus_queue.cc:260] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:25.113727 1296 ts_tablet_manager.cc:1428] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260806 04:00:25.115986 32506 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-d20f5ce3f2ac40379306e3d630f63eda received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:25.116422 32506 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-d20f5ce3f2ac40379306e3d630f63eda on tablet d20f5ce3f2ac40379306e3d630f63eda with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:25.240697 1192 raft_consensus.cc:1203] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260806 04:00:25.404410 1192 raft_consensus.cc:1203] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:25.466398 1192 raft_consensus.cc:1203] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:25.572772 1192 raft_consensus.cc:1203] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:25.769404 1313 raft_consensus.cc:1050] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:25.771961 1313 consensus_queue.cc:237] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:25.779235 1192 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:25.779235 32743 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:25.780849 32608 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:25.780849 948 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:25.782270 1313 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:25.783699 732 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:25.798177 1192 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:25.800628 32609 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:25.796216 1313 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:25.810436 32324 catalog_manager.cc:5495] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:25.824342 32739 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:25.848737 32469 consensus_queue.cc:237] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:25.859167 1192 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:25.860414 948 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:25.860749 32739 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:25.862258 948 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:25.867313 1313 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:25.868902 1192 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:25.871311 32739 raft_consensus.cc:2932] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:25.876267 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3c9b09ac79f84fb797c2a263c612eee5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:25.885591 32321 catalog_manager.cc:5495] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:25.900547 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c9b09ac79f84fb797c2a263c612eee5 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:25.904294 1330 tablet_replica.cc:330] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:25.906245 1330 raft_consensus.cc:2229] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:25.907542 1330 raft_consensus.cc:2258] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:25.912433 1330 ts_tablet_manager.cc:1905] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:25.932664 1330 ts_tablet_manager.cc:1918] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260806 04:00:25.933346 1330 log.cc:1196] T 3c9b09ac79f84fb797c2a263c612eee5 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/3c9b09ac79f84fb797c2a263c612eee5
I20260806 04:00:25.935406 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 3c9b09ac79f84fb797c2a263c612eee5 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:26.000366 998 raft_consensus.cc:1050] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:26.002254 998 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:26.007737 1192 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.007874 32608 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.009615 1311 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:26.012686 1311 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:26.017997 32467 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.019311 1335 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:26.033769 1288 raft_consensus.cc:1050] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:26.034049 1335 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.037204 32608 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.042421 1288 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:26.045763 32467 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.053828 32743 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.056278 1192 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.061655 1315 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:26.062791 1192 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.059252 32321 catalog_manager.cc:5495] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.078244 1313 raft_consensus.cc:1050] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:26.077860 1315 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:26.082204 1313 consensus_queue.cc:237] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:26.106961 1192 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.108927 948 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:26.114557 32745 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.116236 948 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:26.124569 32477 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.125308 32609 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:26.131772 1341 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 04:00:26.142760 1352 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260806 04:00:26.154978 32739 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:26.161573 1192 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:26.165568 998 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:26.173242 32614 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:26.175958 998 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:26.181720 998 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.182444 1192 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.187523 32749 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.184636 1315 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.199569 1352 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.204605 32610 raft_consensus.cc:2932] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.204766 32745 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.223148 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7ec5d99c8efa405b84db49741c2f8933 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:26.229857 32614 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.253322 32321 catalog_manager.cc:5495] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.296399 1190 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.296394 1192 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.309181 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ec5d99c8efa405b84db49741c2f8933 with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:26.306355 32324 catalog_manager.cc:5495] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:26.313705 32324 catalog_manager.cc:5495] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.321308 32469 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.353307 32466 consensus_queue.cc:237] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:26.370131 1363 tablet_replica.cc:330] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:26.374594 1363 raft_consensus.cc:2229] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:26.375691 1363 raft_consensus.cc:2258] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:26.380273 1363 ts_tablet_manager.cc:1905] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:26.380801 32610 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:26.382491 32614 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:26.383013 1192 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:26.393612 32469 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:26.395164 1341 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:26.398371 948 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:26.399401 1192 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:26.401878 1352 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:26.405277 1357 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.410679 1360 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:26.411652 32477 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.415988 1363 ts_tablet_manager.cc:1918] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:26.423564 1363 log.cc:1196] T 7ec5d99c8efa405b84db49741c2f8933 P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/7ec5d99c8efa405b84db49741c2f8933
I20260806 04:00:26.422608 1192 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.425984 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet 7ec5d99c8efa405b84db49741c2f8933 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:26.419380 732 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.430085 1190 raft_consensus.cc:2932] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.441798 32324 catalog_manager.cc:5495] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.436828 32610 raft_consensus.cc:2932] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.454167 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0bf6a30af60042f9b0a84eb0fdb8db19 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:26.464865 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet d20f5ce3f2ac40379306e3d630f63eda with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:26.468561 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d20f5ce3f2ac40379306e3d630f63eda with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:26.472302 32324 catalog_manager.cc:5495] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:26.478143 1365 tablet_replica.cc:330] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:26.479904 1365 raft_consensus.cc:2229] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:26.481067 1365 raft_consensus.cc:2258] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:26.486119 1365 ts_tablet_manager.cc:1905] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:26.494415 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf6a30af60042f9b0a84eb0fdb8db19 with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:26.504685 1365 ts_tablet_manager.cc:1918] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:26.505100 1365 log.cc:1196] T d20f5ce3f2ac40379306e3d630f63eda P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/d20f5ce3f2ac40379306e3d630f63eda
I20260806 04:00:26.506850 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet d20f5ce3f2ac40379306e3d630f63eda (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:26.507061 1365 tablet_replica.cc:330] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:26.508468 1365 raft_consensus.cc:2229] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:26.509121 1365 raft_consensus.cc:2258] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:26.512816 1365 ts_tablet_manager.cc:1905] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:26.529668 1365 ts_tablet_manager.cc:1918] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:26.530236 1365 log.cc:1196] T 0bf6a30af60042f9b0a84eb0fdb8db19 P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/0bf6a30af60042f9b0a84eb0fdb8db19
I20260806 04:00:26.531996 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 0bf6a30af60042f9b0a84eb0fdb8db19 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:27.538125 32610 consensus_queue.cc:237] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:27.550977 32749 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:27.551142 32551 consensus_peers.cc:487] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff861725fdbf4efda0de572d435f5d6d. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:27.551638 32466 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:27.552222 1366 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:27.553047 1360 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:27.561829 1357 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.564230 32466 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.568117 32745 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.584468 32477 consensus_queue.cc:237] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:27.589551 32321 catalog_manager.cc:5495] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
W20260806 04:00:27.597343 32427 consensus_peers.cc:487] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42a553c7b71d493284e5e79b6f19f848. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:27.597450 32610 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:27.599499 32749 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:27.600822 732 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:27.601743 1313 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:27.610834 732 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.612568 32608 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.619181 32749 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.641505 32610 consensus_queue.cc:237] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:27.651713 32749 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:27.653460 32551 consensus_peers.cc:487] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11b58a6c0e4e409fb675ad1e01ef8b60. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:27.655534 32477 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:27.656888 1373 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:27.657869 1357 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:27.658950 32321 catalog_manager.cc:5495] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:27.667666 1373 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.669677 32466 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.671367 32749 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.686192 32745 consensus_queue.cc:237] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:27.686908 32326 catalog_manager.cc:5495] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:27.694371 32466 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:27.694984 32682 consensus_peers.cc:487] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f1e254fda3f4e30802dcd0bba7a82ca. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:27.695866 1342 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:27.697849 32610 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:27.699038 1342 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:27.720018 1342 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.726568 32608 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.727823 32477 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:27.742065 32321 catalog_manager.cc:5495] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:27.978039 1403 ts_tablet_manager.cc:927] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:27.987036 1403 tablet_copy_client.cc:316] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:27.989384 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet ff861725fdbf4efda0de572d435f5d6d from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:27.990070 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet ff861725fdbf4efda0de572d435f5d6d from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-ff861725fdbf4efda0de572d435f5d6d
I20260806 04:00:28.000276 32631 tablet_copy_source_session.cc:205] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:28.004247 1403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff861725fdbf4efda0de572d435f5d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:28.021476 1403 tablet_copy_client.cc:791] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:28.022167 1403 tablet_copy_client.cc:655] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:28.025808 1403 tablet_copy_client.cc:531] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:28.033327 1403 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:28.056273 1403 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:28.057273 1403 tablet_bootstrap.cc:492] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:28.058097 1403 ts_tablet_manager.cc:1397] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.009s
I20260806 04:00:28.061275 1403 raft_consensus.cc:348] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:28.063114 1403 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:28.063861 1403 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:28.067404 1403 ts_tablet_manager.cc:1428] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260806 04:00:28.070479 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-ff861725fdbf4efda0de572d435f5d6d received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:28.070887 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-ff861725fdbf4efda0de572d435f5d6d on tablet ff861725fdbf4efda0de572d435f5d6d with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:28.100386 1403 ts_tablet_manager.cc:927] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:28.102835 1403 tablet_copy_client.cc:316] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:28.104645 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 42a553c7b71d493284e5e79b6f19f848 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:28.105239 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 42a553c7b71d493284e5e79b6f19f848 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-42a553c7b71d493284e5e79b6f19f848
I20260806 04:00:28.112816 32506 tablet_copy_source_session.cc:205] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:28.115859 1403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a553c7b71d493284e5e79b6f19f848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:28.132762 1403 tablet_copy_client.cc:791] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:28.133404 1403 tablet_copy_client.cc:655] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:28.137321 1403 tablet_copy_client.cc:531] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:28.145099 1403 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:28.164589 1403 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:28.165616 1403 tablet_bootstrap.cc:492] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:28.166560 1403 ts_tablet_manager.cc:1397] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.000s
I20260806 04:00:28.169447 1403 raft_consensus.cc:348] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:28.170305 1403 raft_consensus.cc:729] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:28.170851 1403 consensus_queue.cc:260] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:28.172994 1403 ts_tablet_manager.cc:1428] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:00:28.174487 32506 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-42a553c7b71d493284e5e79b6f19f848 received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:28.174880 32506 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-42a553c7b71d493284e5e79b6f19f848 on tablet 42a553c7b71d493284e5e79b6f19f848 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:28.184569 1403 ts_tablet_manager.cc:927] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:28.186170 1403 tablet_copy_client.cc:316] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:28.188165 300 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet 3f1e254fda3f4e30802dcd0bba7a82ca from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:28.188763 300 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet 3f1e254fda3f4e30802dcd0bba7a82ca from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-3f1e254fda3f4e30802dcd0bba7a82ca
I20260806 04:00:28.196784 300 tablet_copy_source_session.cc:205] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:28.206033 1403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1e254fda3f4e30802dcd0bba7a82ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:28.228266 1403 tablet_copy_client.cc:791] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:28.229141 1403 tablet_copy_client.cc:655] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:28.229920 1414 ts_tablet_manager.cc:927] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:28.233290 1403 tablet_copy_client.cc:531] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:28.241125 1414 tablet_copy_client.cc:316] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:28.242628 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 11b58a6c0e4e409fb675ad1e01ef8b60 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:28.243160 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 11b58a6c0e4e409fb675ad1e01ef8b60 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-11b58a6c0e4e409fb675ad1e01ef8b60
I20260806 04:00:28.244556 1403 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:28.250445 32631 tablet_copy_source_session.cc:205] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:28.254477 1414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b58a6c0e4e409fb675ad1e01ef8b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:28.267371 1403 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:28.268626 1403 tablet_bootstrap.cc:492] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:28.270447 1403 ts_tablet_manager.cc:1397] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.008s
I20260806 04:00:28.271008 1414 tablet_copy_client.cc:791] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:28.271571 1414 tablet_copy_client.cc:655] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:28.273926 1403 raft_consensus.cc:348] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:28.274855 1403 raft_consensus.cc:729] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:28.275455 1403 consensus_queue.cc:260] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:28.279091 1414 tablet_copy_client.cc:531] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:28.278825 1403 ts_tablet_manager.cc:1428] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:00:28.283059 300 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-3f1e254fda3f4e30802dcd0bba7a82ca received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:28.283497 300 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-3f1e254fda3f4e30802dcd0bba7a82ca on tablet 3f1e254fda3f4e30802dcd0bba7a82ca with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:28.296461 1414 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:28.314713 1414 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:28.315682 1414 tablet_bootstrap.cc:492] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:28.316452 1414 ts_tablet_manager.cc:1397] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s
I20260806 04:00:28.319248 1414 raft_consensus.cc:348] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:28.320132 1414 raft_consensus.cc:729] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:28.320737 1414 consensus_queue.cc:260] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:28.323174 1414 ts_tablet_manager.cc:1428] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:00:28.327550 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-11b58a6c0e4e409fb675ad1e01ef8b60 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:28.327945 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-11b58a6c0e4e409fb675ad1e01ef8b60 on tablet 11b58a6c0e4e409fb675ad1e01ef8b60 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:28.517568 1190 raft_consensus.cc:1203] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:28.542601 1190 raft_consensus.cc:1203] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:28.609074 1190 raft_consensus.cc:1203] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:28.692768 1190 raft_consensus.cc:1203] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:28.969290 1367 raft_consensus.cc:1050] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:28.971606 1367 consensus_queue.cc:237] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:28.977825 1190 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:28.978046 32749 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:28.979517 32610 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:28.979393 732 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:28.981106 1367 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:28.982154 732 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:28.991281 1367 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:28.992929 1190 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.000120 32610 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.007004 32743 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.010192 1366 raft_consensus.cc:1050] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:29.018716 32321 catalog_manager.cc:5495] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:29.024163 1366 consensus_queue.cc:237] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:29.059087 1190 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.059826 32743 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.063133 32477 consensus_queue.cc:237] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:29.073424 1418 raft_consensus.cc:1050] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:29.067396 1366 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:29.068732 32466 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.081658 1399 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:29.083876 1438 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:29.084467 1190 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:29.086390 1313 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:29.097941 32745 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:29.108812 732 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260806 04:00:29.121646 732 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.140550 1189 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.105958 1418 consensus_queue.cc:237] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:29.168756 1395 raft_consensus.cc:1050] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:29.173677 1395 consensus_queue.cc:237] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:29.174083 32324 catalog_manager.cc:5495] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.177793 1446 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.183611 32477 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.188681 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a553c7b71d493284e5e79b6f19f848 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:29.196024 1456 tablet_replica.cc:330] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:29.186761 1192 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.213351 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 42a553c7b71d493284e5e79b6f19f848 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:29.227123 1189 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.233733 32741 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.231427 32745 raft_consensus.cc:2932] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.237742 32477 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.239447 1366 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:29.241269 1443 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:29.244308 1366 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:29.221402 1456 raft_consensus.cc:2229] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:29.292343 32321 catalog_manager.cc:5495] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.299733 32609 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.304749 1192 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.306381 1451 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:29.293371 1456 raft_consensus.cc:2258] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:29.309315 1450 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 04:00:29.320216 32477 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:29.324657 1366 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.326196 32469 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.329154 1451 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:29.333096 1456 ts_tablet_manager.cc:1905] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:29.358664 32745 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.344333 1190 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.357744 32741 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.377416 1456 ts_tablet_manager.cc:1918] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:29.378039 1451 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.379547 32608 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.379416 32321 catalog_manager.cc:5495] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.389061 32609 consensus_queue.cc:237] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:29.396761 1189 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:29.399979 1366 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:29.378037 1456 log.cc:1196] T 42a553c7b71d493284e5e79b6f19f848 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/42a553c7b71d493284e5e79b6f19f848
I20260806 04:00:29.407799 1189 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.408905 32466 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.412725 32469 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:29.414165 1366 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:29.416316 1446 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.422991 1192 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.431392 32466 raft_consensus.cc:2932] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.431728 32321 catalog_manager.cc:5495] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.441425 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 42a553c7b71d493284e5e79b6f19f848 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:29.455901 32743 consensus_queue.cc:237] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:29.457739 32324 catalog_manager.cc:5495] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.464502 32615 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:29.465889 1395 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:29.467483 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 11b58a6c0e4e409fb675ad1e01ef8b60 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:29.463626 1192 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:29.475172 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b58a6c0e4e409fb675ad1e01ef8b60 with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:29.474985 1395 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:29.482259 1451 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.485926 1192 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.486706 32615 raft_consensus.cc:2932] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.490535 32609 consensus_queue.cc:237] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:29.496609 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3f1e254fda3f4e30802dcd0bba7a82ca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:29.498245 32741 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:29.500554 1446 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:29.501123 1468 tablet_replica.cc:330] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:29.502234 1468 raft_consensus.cc:2229] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:29.503512 1468 raft_consensus.cc:2258] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:29.506042 1189 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:29.512990 1366 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:29.514966 1468 ts_tablet_manager.cc:1905] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:29.511409 32322 catalog_manager.cc:5495] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:29.530000 1443 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.532613 32741 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.551810 1192 raft_consensus.cc:2932] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:29.568589 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ff861725fdbf4efda0de572d435f5d6d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:29.572628 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f1e254fda3f4e30802dcd0bba7a82ca with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:29.573406 1468 ts_tablet_manager.cc:1918] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:29.575802 1468 log.cc:1196] T 11b58a6c0e4e409fb675ad1e01ef8b60 P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/11b58a6c0e4e409fb675ad1e01ef8b60
I20260806 04:00:29.575121 1469 tablet_replica.cc:330] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:29.577422 1469 raft_consensus.cc:2229] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:29.577921 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 11b58a6c0e4e409fb675ad1e01ef8b60 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:29.578155 1469 raft_consensus.cc:2258] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:29.580983 32322 catalog_manager.cc:5495] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:29.586104 1469 ts_tablet_manager.cc:1905] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:29.593384 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet ff861725fdbf4efda0de572d435f5d6d with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:29.606783 1469 ts_tablet_manager.cc:1918] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:29.607178 1469 log.cc:1196] T 3f1e254fda3f4e30802dcd0bba7a82ca P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/3f1e254fda3f4e30802dcd0bba7a82ca
I20260806 04:00:29.608609 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet 3f1e254fda3f4e30802dcd0bba7a82ca (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:29.608734 1469 tablet_replica.cc:330] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:29.609632 1469 raft_consensus.cc:2229] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:29.610210 1469 raft_consensus.cc:2258] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:29.612793 1469 ts_tablet_manager.cc:1905] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:29.624986 1469 ts_tablet_manager.cc:1918] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:29.625391 1469 log.cc:1196] T ff861725fdbf4efda0de572d435f5d6d P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/ff861725fdbf4efda0de572d435f5d6d
I20260806 04:00:29.626693 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet ff861725fdbf4efda0de572d435f5d6d (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:30.666115 32743 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:00:30.672230 32682 consensus_peers.cc:487] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c1c97829f044cfe84bbe23b612c3b81. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:30.672626 32477 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:00:30.673174 32609 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:00:30.673820 1420 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:30.674696 1451 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:30.684922 1420 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.686021 32615 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.693467 32477 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.698935 32743 consensus_queue.cc:237] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:00:30.705904 32682 consensus_peers.cc:487] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45b7481674ca403b9a224047a6d99b03. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:30.707122 32466 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:30.709401 32615 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:30.708765 32326 catalog_manager.cc:5495] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.708747 1451 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260806 04:00:30.716240 1451 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:30.720605 1487 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.729728 32477 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.740614 32609 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.762949 32321 catalog_manager.cc:5495] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.770044 32466 consensus_queue.cc:237] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:30.781467 32743 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:30.781840 32609 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:30.782509 32427 consensus_peers.cc:487] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57c0e905630e43668379f2f95684775f. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:30.783067 1470 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:30.784716 1470 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:30.792639 1454 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.795600 32609 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.794526 32743 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.809300 32741 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:30.815991 32326 catalog_manager.cc:5495] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.823068 32609 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:30.823321 32466 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:30.824393 32682 consensus_peers.cc:487] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65c8a38c9f554ef887e7782aa21ad891. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:30.824731 1487 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:30.826232 1451 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:30.835762 1487 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.839170 32615 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.844341 32466 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:30.863148 32321 catalog_manager.cc:5495] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:31.189548 1516 ts_tablet_manager.cc:927] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:31.191582 1516 tablet_copy_client.cc:316] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:31.193295 300 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet 45b7481674ca403b9a224047a6d99b03 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:31.193813 300 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet 45b7481674ca403b9a224047a6d99b03 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-45b7481674ca403b9a224047a6d99b03
I20260806 04:00:31.202716 300 tablet_copy_source_session.cc:205] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:31.205716 1516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45b7481674ca403b9a224047a6d99b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:31.220750 1516 tablet_copy_client.cc:791] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:31.221411 1516 tablet_copy_client.cc:655] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:31.223101 1518 ts_tablet_manager.cc:927] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:31.225945 1516 tablet_copy_client.cc:531] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:31.227968 1521 ts_tablet_manager.cc:927] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:31.231657 1518 tablet_copy_client.cc:316] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:31.233364 300 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet 3c1c97829f044cfe84bbe23b612c3b81 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:31.233852 300 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet 3c1c97829f044cfe84bbe23b612c3b81 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-3c1c97829f044cfe84bbe23b612c3b81
I20260806 04:00:31.239559 1516 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:31.241307 1521 tablet_copy_client.cc:316] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:31.241710 300 tablet_copy_source_session.cc:205] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:31.243239 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 57c0e905630e43668379f2f95684775f from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:31.243769 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 57c0e905630e43668379f2f95684775f from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-57c0e905630e43668379f2f95684775f
I20260806 04:00:31.250607 32506 tablet_copy_source_session.cc:205] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:31.255827 1518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1c97829f044cfe84bbe23b612c3b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:31.257488 1521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c0e905630e43668379f2f95684775f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:31.268929 1516 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:31.270005 1516 tablet_bootstrap.cc:492] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:31.270736 1516 ts_tablet_manager.cc:1397] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.000s
I20260806 04:00:31.272135 1518 tablet_copy_client.cc:791] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:31.272744 1518 tablet_copy_client.cc:655] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:31.273471 1516 raft_consensus.cc:348] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:31.274477 1516 raft_consensus.cc:729] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:31.275166 1516 consensus_queue.cc:260] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:31.276553 1518 tablet_copy_client.cc:531] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:31.281111 1521 tablet_copy_client.cc:791] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:31.281766 1521 tablet_copy_client.cc:655] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:31.285804 1518 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:31.291076 1516 ts_tablet_manager.cc:1428] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.020s user 0.007s sys 0.008s
I20260806 04:00:31.292877 300 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-45b7481674ca403b9a224047a6d99b03 received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:31.293277 300 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-45b7481674ca403b9a224047a6d99b03 on tablet 45b7481674ca403b9a224047a6d99b03 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:31.296526 1521 tablet_copy_client.cc:531] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:31.306324 1521 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:31.307052 1516 ts_tablet_manager.cc:927] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:31.310055 1516 tablet_copy_client.cc:316] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:31.312680 300 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet 65c8a38c9f554ef887e7782aa21ad891 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:31.313194 300 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet 65c8a38c9f554ef887e7782aa21ad891 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-65c8a38c9f554ef887e7782aa21ad891
I20260806 04:00:31.326930 1521 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:31.327078 300 tablet_copy_source_session.cc:205] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:31.327688 1518 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:31.329056 1518 tablet_bootstrap.cc:492] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:31.327845 1521 tablet_bootstrap.cc:492] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:31.329761 1518 ts_tablet_manager.cc:1397] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.044s user 0.022s sys 0.010s
I20260806 04:00:31.330801 1521 ts_tablet_manager.cc:1397] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.010s
I20260806 04:00:31.331521 1516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c8a38c9f554ef887e7782aa21ad891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:31.332490 1518 raft_consensus.cc:348] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:31.333357 1518 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:31.334241 1521 raft_consensus.cc:348] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:31.335315 1521 raft_consensus.cc:729] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:31.334928 1518 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:31.337684 1518 ts_tablet_manager.cc:1428] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 04:00:31.338469 1521 consensus_queue.cc:260] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:31.341431 1521 ts_tablet_manager.cc:1428] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260806 04:00:31.342687 300 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-3c1c97829f044cfe84bbe23b612c3b81 received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:31.343143 300 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-3c1c97829f044cfe84bbe23b612c3b81 on tablet 3c1c97829f044cfe84bbe23b612c3b81 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:31.343364 32506 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-57c0e905630e43668379f2f95684775f received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:31.343729 32506 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-57c0e905630e43668379f2f95684775f on tablet 57c0e905630e43668379f2f95684775f with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:31.362036 1516 tablet_copy_client.cc:791] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:31.362787 1516 tablet_copy_client.cc:655] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:31.366041 1516 tablet_copy_client.cc:531] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:31.374366 1516 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:31.392839 1516 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:31.393769 1516 tablet_bootstrap.cc:492] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:31.394387 1516 ts_tablet_manager.cc:1397] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
I20260806 04:00:31.396947 1516 raft_consensus.cc:348] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:31.397825 1516 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:31.398554 1516 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:31.400878 1516 ts_tablet_manager.cc:1428] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:00:31.402740 300 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-65c8a38c9f554ef887e7782aa21ad891 received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:31.403174 300 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-65c8a38c9f554ef887e7782aa21ad891 on tablet 65c8a38c9f554ef887e7782aa21ad891 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:31.654822 1189 raft_consensus.cc:1203] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:31.745070 1189 raft_consensus.cc:1203] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20260806 04:00:31.813297 1189 raft_consensus.cc:1203] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:31.855854 1189 raft_consensus.cc:1203] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:32.171538 1470 raft_consensus.cc:1050] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:32.173684 1470 consensus_queue.cc:237] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:32.179608 1189 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.181593 32610 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.182576 32745 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.183491 1454 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:32.185268 1454 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:32.186769 1535 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:32.217151 1470 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.228814 32610 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.237017 32322 catalog_manager.cc:5495] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:32.238584 1189 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.244855 1487 raft_consensus.cc:1050] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:32.247308 1487 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:32.224831 32741 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.276718 1189 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:00:32.277056 32469 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:00:32.278280 1532 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:32.279323 32609 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:00:32.280720 1451 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:32.282260 1451 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:32.305464 1451 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.307401 32609 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.309762 32466 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.316638 1192 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.335278 32469 consensus_queue.cc:237] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:32.338891 32322 catalog_manager.cc:5495] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.342025 1189 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:32.342969 32610 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:32.344841 1535 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:32.346599 1535 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:32.357013 32741 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:32.357139 1454 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.362334 32610 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.363919 1189 raft_consensus.cc:2932] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.366205 32469 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:00:32.367646 1532 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:32.367996 1192 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:00:32.370323 1532 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:32.373611 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 57c0e905630e43668379f2f95684775f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:32.378566 32477 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.383599 32324 catalog_manager.cc:5495] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.377440 1487 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.394800 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3c1c97829f044cfe84bbe23b612c3b81 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:32.405260 32322 catalog_manager.cc:5495] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:32.402719 1189 raft_consensus.cc:2932] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.409837 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c0e905630e43668379f2f95684775f with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:32.422859 1532 raft_consensus.cc:1050] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:32.426921 1532 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:32.433917 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c1c97829f044cfe84bbe23b612c3b81 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:32.436617 1560 tablet_replica.cc:330] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:32.437438 1560 raft_consensus.cc:2229] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:32.438097 1560 raft_consensus.cc:2258] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:32.439391 1420 raft_consensus.cc:1050] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:32.441977 1560 ts_tablet_manager.cc:1905] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:32.442437 1420 consensus_queue.cc:237] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:32.448654 32609 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.449144 32477 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.451745 1532 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260806 04:00:32.456959 1487 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:32.460554 1192 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.461419 32609 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.462931 1554 tablet_replica.cc:330] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:32.463261 32466 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.458781 1189 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:32.464135 1487 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260806 04:00:32.481415 1451 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:32.483657 1420 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:32.477933 1532 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 04:00:32.464099 1554 raft_consensus.cc:2229] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:32.488030 1554 raft_consensus.cc:2258] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:32.520869 1554 ts_tablet_manager.cc:1905] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:32.529965 1560 ts_tablet_manager.cc:1918] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260806 04:00:32.543876 1560 log.cc:1196] T 3c1c97829f044cfe84bbe23b612c3b81 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/3c1c97829f044cfe84bbe23b612c3b81
I20260806 04:00:32.545756 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 3c1c97829f044cfe84bbe23b612c3b81 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:32.547433 1554 ts_tablet_manager.cc:1918] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:32.547896 1554 log.cc:1196] T 57c0e905630e43668379f2f95684775f P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/57c0e905630e43668379f2f95684775f
I20260806 04:00:32.549805 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 57c0e905630e43668379f2f95684775f (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:32.552670 1532 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.555212 1487 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.555975 32477 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.556684 32466 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.567551 32609 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.579780 1192 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.580014 1189 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.574718 32321 catalog_manager.cc:5495] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.583369 32615 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.600437 32321 catalog_manager.cc:5495] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.640350 32745 consensus_queue.cc:237] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:32.640640 32741 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:32.645246 1189 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:32.646030 32609 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:32.647307 32477 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:32.647408 1189 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:32.648780 1577 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:32.649428 1451 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:32.650372 1577 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:32.650326 1564 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:32.657375 1577 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.659705 32477 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.663579 1189 raft_consensus.cc:2932] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.666857 1190 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.669588 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 65c8a38c9f554ef887e7782aa21ad891 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:32.675253 1420 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.679581 32609 raft_consensus.cc:2932] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.690367 32326 catalog_manager.cc:5495] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.692930 32324 catalog_manager.cc:5495] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:32.705798 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 45b7481674ca403b9a224047a6d99b03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:32.707813 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c8a38c9f554ef887e7782aa21ad891 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:32.708880 1560 tablet_replica.cc:330] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:32.710548 1560 raft_consensus.cc:2229] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:32.711385 1560 raft_consensus.cc:2258] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:32.714898 1560 ts_tablet_manager.cc:1905] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:32.717217 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet 45b7481674ca403b9a224047a6d99b03 with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:32.734391 1578 tablet_replica.cc:330] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:32.736699 1578 raft_consensus.cc:2229] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:32.737695 1578 raft_consensus.cc:2258] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:32.741389 1560 ts_tablet_manager.cc:1918] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:32.742069 1560 log.cc:1196] T 65c8a38c9f554ef887e7782aa21ad891 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/65c8a38c9f554ef887e7782aa21ad891
I20260806 04:00:32.744220 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 65c8a38c9f554ef887e7782aa21ad891 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:32.746294 1578 ts_tablet_manager.cc:1905] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:32.766651 1578 ts_tablet_manager.cc:1918] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:32.767163 1578 log.cc:1196] T 45b7481674ca403b9a224047a6d99b03 P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/45b7481674ca403b9a224047a6d99b03
I20260806 04:00:32.770264 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet 45b7481674ca403b9a224047a6d99b03 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:33.770725 32477 consensus_queue.cc:237] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:33.777714 32609 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:33.778242 32427 consensus_peers.cc:487] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46d78e8748804374b8af237c1ca8ede8. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:33.778546 32741 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:33.779397 1580 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:33.780448 1583 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:33.788338 1470 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.789886 32745 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.804052 32321 catalog_manager.cc:5495] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:33.804183 32610 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.804977 32477 consensus_queue.cc:237] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:33.839084 32609 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:33.839195 32427 consensus_peers.cc:487] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bc7e4a68014409c8199e1e003add00a. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:33.841109 32745 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:33.842079 1583 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:33.843189 1580 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:33.854048 1583 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.854957 32745 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.869076 32615 consensus_queue.cc:237] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:33.881500 32324 catalog_manager.cc:5495] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 04:00:33.894148 32551 consensus_peers.cc:487] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec440a6dfdfc46af8a0876b464ef531d. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:33.894042 32609 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.898483 32467 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:33.898612 32749 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:33.900035 1443 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:33.900913 1472 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:33.911031 1443 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.916795 32467 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.930543 32745 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.935127 32609 consensus_queue.cc:237] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:33.936278 32324 catalog_manager.cc:5495] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.944763 32749 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:33.945309 32551 consensus_peers.cc:487] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 725b0e6610ef466d8d7cf9ed945d2d4e. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:33.945375 32467 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:33.946195 1605 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:33.949149 1472 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:33.961177 1472 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.963034 32745 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.969137 32467 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:33.987298 32321 catalog_manager.cc:5495] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:34.326058 1615 ts_tablet_manager.cc:927] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:34.328513 1615 tablet_copy_client.cc:316] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:34.330024 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 46d78e8748804374b8af237c1ca8ede8 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:34.330503 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 46d78e8748804374b8af237c1ca8ede8 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-46d78e8748804374b8af237c1ca8ede8
I20260806 04:00:34.338276 32506 tablet_copy_source_session.cc:205] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:34.343101 1617 ts_tablet_manager.cc:927] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:34.344511 1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d78e8748804374b8af237c1ca8ede8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:34.346763 1617 tablet_copy_client.cc:316] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:34.348300 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 725b0e6610ef466d8d7cf9ed945d2d4e from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:34.348883 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 725b0e6610ef466d8d7cf9ed945d2d4e from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-725b0e6610ef466d8d7cf9ed945d2d4e
I20260806 04:00:34.361768 1615 tablet_copy_client.cc:791] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:34.362486 1615 tablet_copy_client.cc:655] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:34.365451 32631 tablet_copy_source_session.cc:205] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:34.366285 1615 tablet_copy_client.cc:531] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:34.368971 1617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725b0e6610ef466d8d7cf9ed945d2d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:34.381162 1621 ts_tablet_manager.cc:927] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:34.385327 1615 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:34.392912 1617 tablet_copy_client.cc:791] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:34.393569 1617 tablet_copy_client.cc:655] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:34.393739 1621 tablet_copy_client.cc:316] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:34.397666 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 0bc7e4a68014409c8199e1e003add00a from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:34.398205 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 0bc7e4a68014409c8199e1e003add00a from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-0bc7e4a68014409c8199e1e003add00a
I20260806 04:00:34.398850 1617 tablet_copy_client.cc:531] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:34.408775 1617 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:34.433830 32506 tablet_copy_source_session.cc:205] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:34.443521 1617 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:34.445185 1617 tablet_bootstrap.cc:492] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:34.445667 1621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc7e4a68014409c8199e1e003add00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:34.446120 1617 ts_tablet_manager.cc:1397] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.000s
I20260806 04:00:34.448966 1617 raft_consensus.cc:348] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:34.449939 1617 raft_consensus.cc:729] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:34.450649 1617 consensus_queue.cc:260] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:34.482564 1621 tablet_copy_client.cc:791] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:34.488008 1621 tablet_copy_client.cc:655] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:34.484755 1615 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:34.490222 1615 tablet_bootstrap.cc:492] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:34.493594 1621 tablet_copy_client.cc:531] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:34.495214 1615 ts_tablet_manager.cc:1397] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.110s user 0.055s sys 0.025s
I20260806 04:00:34.481642 1630 ts_tablet_manager.cc:927] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:34.483402 1617 ts_tablet_manager.cc:1428] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.037s user 0.023s sys 0.012s
I20260806 04:00:34.503146 1621 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:34.503108 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-725b0e6610ef466d8d7cf9ed945d2d4e received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:34.503655 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-725b0e6610ef466d8d7cf9ed945d2d4e on tablet 725b0e6610ef466d8d7cf9ed945d2d4e with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:34.501363 1615 raft_consensus.cc:348] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:34.504225 1615 raft_consensus.cc:729] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:34.504873 1615 consensus_queue.cc:260] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:34.507293 1615 ts_tablet_manager.cc:1428] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260806 04:00:34.510485 32506 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-46d78e8748804374b8af237c1ca8ede8 received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:34.510864 32506 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-46d78e8748804374b8af237c1ca8ede8 on tablet 46d78e8748804374b8af237c1ca8ede8 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:34.520316 1630 tablet_copy_client.cc:316] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:34.523114 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet ec440a6dfdfc46af8a0876b464ef531d from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:34.523663 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet ec440a6dfdfc46af8a0876b464ef531d from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-ec440a6dfdfc46af8a0876b464ef531d
I20260806 04:00:34.531431 32631 tablet_copy_source_session.cc:205] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:34.534226 1630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec440a6dfdfc46af8a0876b464ef531d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:34.536156 1621 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:34.536958 1621 tablet_bootstrap.cc:492] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:34.537571 1621 ts_tablet_manager.cc:1397] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.014s
I20260806 04:00:34.539654 1621 raft_consensus.cc:348] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:34.540361 1621 raft_consensus.cc:729] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:34.541010 1621 consensus_queue.cc:260] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:34.543756 1621 ts_tablet_manager.cc:1428] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:00:34.545219 32506 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-0bc7e4a68014409c8199e1e003add00a received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:34.545589 32506 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-0bc7e4a68014409c8199e1e003add00a on tablet 0bc7e4a68014409c8199e1e003add00a with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:34.549824 1630 tablet_copy_client.cc:791] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:34.550407 1630 tablet_copy_client.cc:655] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:34.558017 1630 tablet_copy_client.cc:531] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:34.565889 1630 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:34.583811 1630 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:34.584609 1630 tablet_bootstrap.cc:492] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:34.585249 1630 ts_tablet_manager.cc:1397] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.006s
I20260806 04:00:34.587685 1630 raft_consensus.cc:348] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:34.588534 1630 raft_consensus.cc:729] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:34.589138 1630 consensus_queue.cc:260] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:34.591365 1630 ts_tablet_manager.cc:1428] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:00:34.593820 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-ec440a6dfdfc46af8a0876b464ef531d received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:34.594112 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-ec440a6dfdfc46af8a0876b464ef531d on tablet ec440a6dfdfc46af8a0876b464ef531d with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:34.709475 1189 raft_consensus.cc:1203] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:34.753216 1189 raft_consensus.cc:1203] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:34.881585 1189 raft_consensus.cc:1203] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:35.038268 1189 raft_consensus.cc:1203] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:35.163893 1470 raft_consensus.cc:1050] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:35.165832 1470 consensus_queue.cc:237] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:35.179618 32745 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.179901 1189 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.184680 1637 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:35.185794 32615 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.187463 1627 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:35.188336 1637 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:35.308447 1643 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.391539 1472 raft_consensus.cc:1050] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:35.394052 1472 consensus_queue.cc:237] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:35.395416 32609 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:35.401391 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 64200us
I20260806 04:00:35.402212 1192 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.407840 1188 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.418058 1662 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:35.420117 32745 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.423039 32469 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.423352 32743 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.429543 1472 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 04:00:35.431583 32322 catalog_manager.cc:5495] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:35.444535 1662 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:35.453964 1626 raft_consensus.cc:1050] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:35.456813 1626 consensus_queue.cc:237] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:35.473013 1662 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.474648 32469 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.477753 1192 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.479334 32615 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.480104 32749 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.481547 1657 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:35.489982 1659 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:35.491218 1650 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:35.495749 32745 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.505157 32469 consensus_queue.cc:237] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:35.509456 1188 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.512861 1189 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:35.517578 32749 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:35.518918 1657 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:35.522174 32322 catalog_manager.cc:5495] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.521325 1626 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.529343 1192 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.539153 1657 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:35.546694 1472 raft_consensus.cc:1050] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:35.546587 1643 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.549095 1472 consensus_queue.cc:237] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:35.551294 32745 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.558990 32743 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.558633 32604 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.563227 32614 consensus_queue.cc:237] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:35.573381 32749 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:35.574748 1443 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:35.578701 1189 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:35.586987 1605 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:35.598404 1646 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.600178 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 46d78e8748804374b8af237c1ca8ede8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:35.603897 1188 raft_consensus.cc:2932] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.602798 32322 catalog_manager.cc:5495] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.610328 32743 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.619053 1190 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.620267 32749 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.622869 1656 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:35.631690 32469 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:35.616714 1189 raft_consensus.cc:2932] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.633863 32324 catalog_manager.cc:5495] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:35.633396 1443 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:35.636615 1472 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:35.648291 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 725b0e6610ef466d8d7cf9ed945d2d4e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:35.659339 32322 catalog_manager.cc:5495] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:35.663921 32477 consensus_queue.cc:237] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:35.687860 1189 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:35.688228 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d78e8748804374b8af237c1ca8ede8 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:35.693776 32614 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:35.691825 1626 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260806 04:00:35.699536 1657 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:35.724406 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet 725b0e6610ef466d8d7cf9ed945d2d4e with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:35.728304 1627 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.756181 1691 tablet_replica.cc:330] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:35.758700 1691 raft_consensus.cc:2229] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:35.770319 1691 raft_consensus.cc:2258] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:35.734272 1188 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.732149 32604 raft_consensus.cc:2932] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.759298 1443 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.771672 1692 tablet_replica.cc:330] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:35.761756 1192 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.776435 1692 raft_consensus.cc:2229] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:35.775867 32749 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.777226 1692 raft_consensus.cc:2258] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:35.764298 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0bc7e4a68014409c8199e1e003add00a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:35.781622 1691 ts_tablet_manager.cc:1905] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:35.794301 1692 ts_tablet_manager.cc:1905] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:35.798590 32469 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.800392 32321 catalog_manager.cc:5495] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:35.795167 32322 catalog_manager.cc:5495] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.845726 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bc7e4a68014409c8199e1e003add00a with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:35.866359 32604 consensus_queue.cc:237] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:35.874475 1691 ts_tablet_manager.cc:1918] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:35.875300 1691 log.cc:1196] T 46d78e8748804374b8af237c1ca8ede8 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/46d78e8748804374b8af237c1ca8ede8
I20260806 04:00:35.876857 1699 tablet_replica.cc:330] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:35.878211 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 46d78e8748804374b8af237c1ca8ede8 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:35.881778 1692 ts_tablet_manager.cc:1918] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:35.883080 1692 log.cc:1196] T 725b0e6610ef466d8d7cf9ed945d2d4e P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/725b0e6610ef466d8d7cf9ed945d2d4e
I20260806 04:00:35.883481 32745 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:35.885205 1188 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:35.885555 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet 725b0e6610ef466d8d7cf9ed945d2d4e (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:35.887964 1656 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:35.889070 1443 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:35.879524 1699 raft_consensus.cc:2229] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:35.897946 1662 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.899744 32749 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.905331 1699 raft_consensus.cc:2258] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:35.909014 1188 raft_consensus.cc:2932] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.910460 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ec440a6dfdfc46af8a0876b464ef531d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:35.910960 1699 ts_tablet_manager.cc:1905] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:35.926043 32322 catalog_manager.cc:5495] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:35.929471 1699 ts_tablet_manager.cc:1918] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:35.930027 1699 log.cc:1196] T 0bc7e4a68014409c8199e1e003add00a P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/0bc7e4a68014409c8199e1e003add00a
I20260806 04:00:35.931821 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 0bc7e4a68014409c8199e1e003add00a (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:35.940250 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet ec440a6dfdfc46af8a0876b464ef531d with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:35.941250 1692 tablet_replica.cc:330] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:35.942512 1692 raft_consensus.cc:2229] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:35.943738 1692 raft_consensus.cc:2258] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:35.947180 1692 ts_tablet_manager.cc:1905] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:35.976809 1692 ts_tablet_manager.cc:1918] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:35.977407 1692 log.cc:1196] T ec440a6dfdfc46af8a0876b464ef531d P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/ec440a6dfdfc46af8a0876b464ef531d
I20260806 04:00:35.979485 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet ec440a6dfdfc46af8a0876b464ef531d (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:37.008301 32467 consensus_queue.cc:237] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.015173 32749 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:37.016983 1657 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:37.017679 32615 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:37.018646 32427 consensus_peers.cc:487] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62779f9120ed4ef3b0d9febc7c2cd224. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:37.020069 1657 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:37.026932 1626 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.029788 32614 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.029409 32749 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.045548 32324 catalog_manager.cc:5495] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:37.046334 32467 consensus_queue.cc:237] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:00:37.061151 32427 consensus_peers.cc:487] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 128baf4b99ba44f0aa14ba803db9e1ba. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:37.061637 32749 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:37.062506 32615 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:37.064675 1657 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:37.069751 1706 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:37.084929 1714 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.086313 32749 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.096925 32615 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.103511 32614 consensus_queue.cc:237] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.110469 32326 catalog_manager.cc:5495] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
W20260806 04:00:37.116845 32551 consensus_peers.cc:487] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c662c1a70954272bb9eff4a51caaeee. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:37.118131 32466 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:37.119035 32749 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:37.122303 1472 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:37.123112 1662 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:37.140677 32743 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.138046 1723 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.145552 32466 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.161742 32322 catalog_manager.cc:5495] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.170063 32743 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:00:37.200724 32682 consensus_peers.cc:487] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe6ca7a29c2e4741a8285425bb994584. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:37.203274 32467 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:37.205121 32615 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:37.205302 1651 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:37.210631 1651 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:37.214403 1704 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.215725 32467 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.221346 32614 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.230865 32322 catalog_manager.cc:5495] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:37.555361 1735 ts_tablet_manager.cc:927] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:37.557456 1735 tablet_copy_client.cc:316] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:37.558859 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 62779f9120ed4ef3b0d9febc7c2cd224 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:37.559376 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 62779f9120ed4ef3b0d9febc7c2cd224 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-62779f9120ed4ef3b0d9febc7c2cd224
I20260806 04:00:37.566540 32506 tablet_copy_source_session.cc:205] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:37.569700 1735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62779f9120ed4ef3b0d9febc7c2cd224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:37.584793 1735 tablet_copy_client.cc:791] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:37.585364 1735 tablet_copy_client.cc:655] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:37.588521 1735 tablet_copy_client.cc:531] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:37.595423 1735 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:37.620301 1744 ts_tablet_manager.cc:927] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:37.622277 1735 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:37.623322 1735 tablet_bootstrap.cc:492] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:37.624130 1735 ts_tablet_manager.cc:1397] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.000s
I20260806 04:00:37.626835 1735 raft_consensus.cc:348] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.627775 1735 raft_consensus.cc:729] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:37.628804 1744 tablet_copy_client.cc:316] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:37.628505 1735 consensus_queue.cc:260] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.630664 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 5c662c1a70954272bb9eff4a51caaeee from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:37.631194 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 5c662c1a70954272bb9eff4a51caaeee from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-5c662c1a70954272bb9eff4a51caaeee
I20260806 04:00:37.638743 1735 ts_tablet_manager.cc:1428] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260806 04:00:37.643286 32506 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-62779f9120ed4ef3b0d9febc7c2cd224 received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:37.643709 32506 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-62779f9120ed4ef3b0d9febc7c2cd224 on tablet 62779f9120ed4ef3b0d9febc7c2cd224 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:37.647152 32631 tablet_copy_source_session.cc:205] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:37.656936 1735 ts_tablet_manager.cc:927] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:37.659376 1744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c662c1a70954272bb9eff4a51caaeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:37.660434 1735 tablet_copy_client.cc:316] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:37.661966 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 128baf4b99ba44f0aa14ba803db9e1ba from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:37.662452 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 128baf4b99ba44f0aa14ba803db9e1ba from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-128baf4b99ba44f0aa14ba803db9e1ba
I20260806 04:00:37.669828 32506 tablet_copy_source_session.cc:205] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:37.672871 1735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128baf4b99ba44f0aa14ba803db9e1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:37.685743 1744 tablet_copy_client.cc:791] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:37.686383 1744 tablet_copy_client.cc:655] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:37.687664 1735 tablet_copy_client.cc:791] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:37.688242 1735 tablet_copy_client.cc:655] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:37.690166 1744 tablet_copy_client.cc:531] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:37.693979 1735 tablet_copy_client.cc:531] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:37.700929 1744 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:37.702957 1735 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:37.723444 1735 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:37.724169 1735 tablet_bootstrap.cc:492] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:37.724733 1735 ts_tablet_manager.cc:1397] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.004s
I20260806 04:00:37.724733 1744 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:37.726203 1744 tablet_bootstrap.cc:492] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:37.726959 1744 ts_tablet_manager.cc:1397] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.013s
I20260806 04:00:37.727843 1735 raft_consensus.cc:348] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.728832 1735 raft_consensus.cc:729] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:37.729315 1735 consensus_queue.cc:260] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.729763 1744 raft_consensus.cc:348] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.730738 1744 raft_consensus.cc:729] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:37.731907 1735 ts_tablet_manager.cc:1428] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:00:37.731467 1744 consensus_queue.cc:260] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.734717 1744 ts_tablet_manager.cc:1428] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 04:00:37.736214 1748 ts_tablet_manager.cc:927] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:37.737663 32506 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-128baf4b99ba44f0aa14ba803db9e1ba received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:37.738098 32506 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-128baf4b99ba44f0aa14ba803db9e1ba on tablet 128baf4b99ba44f0aa14ba803db9e1ba with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:37.739423 1748 tablet_copy_client.cc:316] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:37.740181 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-5c662c1a70954272bb9eff4a51caaeee received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:37.740603 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-5c662c1a70954272bb9eff4a51caaeee on tablet 5c662c1a70954272bb9eff4a51caaeee with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:37.740942 300 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet fe6ca7a29c2e4741a8285425bb994584 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:37.741484 300 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet fe6ca7a29c2e4741a8285425bb994584 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-fe6ca7a29c2e4741a8285425bb994584
I20260806 04:00:37.751006 300 tablet_copy_source_session.cc:205] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:37.754091 1748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6ca7a29c2e4741a8285425bb994584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:37.767210 1748 tablet_copy_client.cc:791] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:37.767836 1748 tablet_copy_client.cc:655] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:37.774861 1748 tablet_copy_client.cc:531] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:37.787531 1748 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:37.806555 1748 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:37.807490 1748 tablet_bootstrap.cc:492] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:37.808171 1748 ts_tablet_manager.cc:1397] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.001s
I20260806 04:00:37.810642 1748 raft_consensus.cc:348] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.811383 1748 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:37.811980 1748 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:37.814755 1748 ts_tablet_manager.cc:1428] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:00:37.816251 300 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-fe6ca7a29c2e4741a8285425bb994584 received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:37.816645 300 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-fe6ca7a29c2e4741a8285425bb994584 on tablet fe6ca7a29c2e4741a8285425bb994584 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:37.945091 1189 raft_consensus.cc:1203] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:38.019322 1189 raft_consensus.cc:1203] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:38.047194 1188 raft_consensus.cc:1203] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:38.127604 1189 raft_consensus.cc:1203] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:38.557168 1657 raft_consensus.cc:1050] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:38.559909 1657 consensus_queue.cc:237] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:38.567555 1188 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.567775 32614 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.570194 32749 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.569878 1749 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:38.572017 1714 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:38.572494 1749 raft_consensus.cc:1050] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:38.574945 1749 consensus_queue.cc:237] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:38.581589 1761 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:38.596544 1188 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.598119 1761 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:38.600814 1654 raft_consensus.cc:1050] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:38.602989 1654 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:38.604701 32749 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.608822 1714 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:38.633339 32615 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.637377 1761 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:38.639292 1714 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.643690 1729 raft_consensus.cc:1050] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:38.646373 1729 consensus_queue.cc:237] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:38.640720 32743 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.666495 1189 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.670017 32469 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.672327 32741 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.672195 1723 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:38.673422 32614 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.674582 1188 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.677222 32322 catalog_manager.cc:5495] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:38.679593 1723 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:38.682783 1723 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:38.718129 1714 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.738663 32745 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.747421 1757 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.752354 1189 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.753531 32469 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.765640 32614 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.765795 1188 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.766729 32466 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:38.768435 1651 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:38.770323 1651 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:38.771978 1651 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 04:00:38.754356 32741 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.793084 1651 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.797767 32466 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.804513 1188 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.808949 32614 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.741430 32615 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.816009 1192 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.848397 32469 consensus_queue.cc:237] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:38.855157 1188 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:38.856570 1749 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:38.860774 32749 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:38.862530 1749 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:38.859069 32324 catalog_manager.cc:5495] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:38.869935 32324 catalog_manager.cc:5495] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.872933 1770 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.874442 1188 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.871479 32324 catalog_manager.cc:5495] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.884886 32741 raft_consensus.cc:2932] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.888375 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 62779f9120ed4ef3b0d9febc7c2cd224 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:38.914199 32469 consensus_queue.cc:237] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:38.917142 32615 consensus_queue.cc:237] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:38.920766 1188 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:38.921164 32614 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:38.922148 1714 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:38.929134 32322 catalog_manager.cc:5495] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.932578 1657 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:38.933065 32749 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:38.933837 1189 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:38.936491 1714 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.941411 1777 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:38.948863 1192 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.949419 1729 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.953243 32741 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:38.955024 1757 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:38.955929 32615 raft_consensus.cc:2932] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.965023 32466 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:38.964506 1189 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.967439 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 128baf4b99ba44f0aa14ba803db9e1ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:38.968976 1651 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:38.970842 32749 raft_consensus.cc:2932] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.976776 1188 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:38.978307 1651 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:38.983770 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 62779f9120ed4ef3b0d9febc7c2cd224 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:38.989393 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5c662c1a70954272bb9eff4a51caaeee with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:38.990479 32322 catalog_manager.cc:5495] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:38.994632 1803 tablet_replica.cc:330] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:38.995711 1803 raft_consensus.cc:2229] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:38.996480 1803 raft_consensus.cc:2258] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:38.997668 32466 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:38.998539 1192 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:39.000751 1803 ts_tablet_manager.cc:1905] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:38.993244 1651 raft_consensus.cc:2932] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:39.009167 32326 catalog_manager.cc:5495] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:39.014238 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 128baf4b99ba44f0aa14ba803db9e1ba with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:39.020706 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c662c1a70954272bb9eff4a51caaeee with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:39.033926 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet fe6ca7a29c2e4741a8285425bb994584 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:39.038719 1806 tablet_replica.cc:330] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:39.040993 1806 raft_consensus.cc:2229] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:39.041985 1806 raft_consensus.cc:2258] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:39.047159 32321 catalog_manager.cc:5495] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:39.057049 1803 ts_tablet_manager.cc:1918] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:39.057922 1803 log.cc:1196] T 62779f9120ed4ef3b0d9febc7c2cd224 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/62779f9120ed4ef3b0d9febc7c2cd224
I20260806 04:00:39.059139 1805 tablet_replica.cc:330] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:39.060289 1805 raft_consensus.cc:2229] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:39.060973 1805 raft_consensus.cc:2258] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:39.061887 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet fe6ca7a29c2e4741a8285425bb994584 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:39.066371 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 62779f9120ed4ef3b0d9febc7c2cd224 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:39.066817 1803 tablet_replica.cc:330] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:39.067821 1803 raft_consensus.cc:2229] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:39.068460 1803 raft_consensus.cc:2258] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:39.069665 1806 ts_tablet_manager.cc:1905] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:39.070374 1805 ts_tablet_manager.cc:1905] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:39.074509 1803 ts_tablet_manager.cc:1905] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:39.090900 1803 ts_tablet_manager.cc:1918] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:39.091396 1803 log.cc:1196] T fe6ca7a29c2e4741a8285425bb994584 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/fe6ca7a29c2e4741a8285425bb994584
I20260806 04:00:39.093107 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet fe6ca7a29c2e4741a8285425bb994584 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:39.099346 1805 ts_tablet_manager.cc:1918] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:39.099815 1805 log.cc:1196] T 128baf4b99ba44f0aa14ba803db9e1ba P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/128baf4b99ba44f0aa14ba803db9e1ba
I20260806 04:00:39.101644 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 128baf4b99ba44f0aa14ba803db9e1ba (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:39.103013 1806 ts_tablet_manager.cc:1918] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:39.103369 1806 log.cc:1196] T 5c662c1a70954272bb9eff4a51caaeee P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/5c662c1a70954272bb9eff4a51caaeee
I20260806 04:00:39.104961 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet 5c662c1a70954272bb9eff4a51caaeee (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:40.108506 32614 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:00:40.115384 32551 consensus_peers.cc:487] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9fdfcc3345c4c86a7d974030c67c271. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:40.115872 32743 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:40.116369 32469 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:40.117334 1729 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:40.118760 1809 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:40.128715 1809 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.136025 32743 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.142798 32469 consensus_queue.cc:237] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.144690 32466 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.147089 32322 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 04:00:40.163355 32427 consensus_peers.cc:487] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e2f839ac9f94c2f980cc895f641607f. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:40.163369 32741 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:40.172688 1769 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:40.175161 32614 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:40.177640 1769 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:40.189330 1714 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.198155 32615 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.203882 32614 consensus_queue.cc:237] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.202805 32743 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.246878 32326 catalog_manager.cc:5495] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
W20260806 04:00:40.248413 32551 consensus_peers.cc:487] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 305cbd07ce124048b7ac2f99b2aa40db. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:40.248368 32743 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:40.250728 32469 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:40.251952 1813 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:40.252892 1729 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:40.267200 1813 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.268851 32469 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.271950 32743 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.282399 32469 consensus_queue.cc:237] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.284907 32326 catalog_manager.cc:5495] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:40.294510 32614 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:40.295239 32743 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:40.295306 32427 consensus_peers.cc:487] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 187160a8bf734a23b90fcc49bc24b10b. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:40.296602 1822 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:40.299726 1714 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:40.308183 1714 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.309267 32743 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.321911 32614 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:40.330054 32321 catalog_manager.cc:5495] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:40.631153 1839 ts_tablet_manager.cc:927] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:40.633491 1839 tablet_copy_client.cc:316] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:40.635177 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 305cbd07ce124048b7ac2f99b2aa40db from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:40.635681 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 305cbd07ce124048b7ac2f99b2aa40db from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-305cbd07ce124048b7ac2f99b2aa40db
I20260806 04:00:40.642356 32631 tablet_copy_source_session.cc:205] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:40.645068 1839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 305cbd07ce124048b7ac2f99b2aa40db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:40.664143 1839 tablet_copy_client.cc:791] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:40.664908 1839 tablet_copy_client.cc:655] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:40.668870 1839 tablet_copy_client.cc:531] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:40.676820 1839 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:40.687080 1841 ts_tablet_manager.cc:927] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:40.689213 1841 tablet_copy_client.cc:316] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:40.690749 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet f9fdfcc3345c4c86a7d974030c67c271 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:40.691272 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet f9fdfcc3345c4c86a7d974030c67c271 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-f9fdfcc3345c4c86a7d974030c67c271
I20260806 04:00:40.704173 1842 ts_tablet_manager.cc:927] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:40.705915 32631 tablet_copy_source_session.cc:205] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:40.709245 1841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fdfcc3345c4c86a7d974030c67c271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:40.718652 1839 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:40.720372 1839 tablet_bootstrap.cc:492] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:40.721078 1839 ts_tablet_manager.cc:1397] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.044s user 0.018s sys 0.009s
I20260806 04:00:40.723794 1846 ts_tablet_manager.cc:927] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:40.724984 1839 raft_consensus.cc:348] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.725965 1839 raft_consensus.cc:729] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:40.726773 1842 tablet_copy_client.cc:316] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:40.726580 1839 consensus_queue.cc:260] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.728503 1846 tablet_copy_client.cc:316] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:40.728777 32506 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 187160a8bf734a23b90fcc49bc24b10b from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:40.729293 32506 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 187160a8bf734a23b90fcc49bc24b10b from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-187160a8bf734a23b90fcc49bc24b10b
I20260806 04:00:40.731863 1841 tablet_copy_client.cc:791] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:40.732554 1841 tablet_copy_client.cc:655] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:40.732906 32505 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 1e2f839ac9f94c2f980cc895f641607f from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:40.733461 32505 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 1e2f839ac9f94c2f980cc895f641607f from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-1e2f839ac9f94c2f980cc895f641607f
I20260806 04:00:40.737812 1839 ts_tablet_manager.cc:1428] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20260806 04:00:40.744406 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-305cbd07ce124048b7ac2f99b2aa40db received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:40.744817 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-305cbd07ce124048b7ac2f99b2aa40db on tablet 305cbd07ce124048b7ac2f99b2aa40db with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:40.748476 32506 tablet_copy_source_session.cc:205] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:40.758020 1841 tablet_copy_client.cc:531] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:40.761591 1842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187160a8bf734a23b90fcc49bc24b10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:40.763723 32505 tablet_copy_source_session.cc:205] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:40.766585 1846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2f839ac9f94c2f980cc895f641607f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:40.796706 1842 tablet_copy_client.cc:791] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:40.797278 1842 tablet_copy_client.cc:655] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:40.800419 1841 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:40.802349 1842 tablet_copy_client.cc:531] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:40.810720 1842 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:40.816018 1846 tablet_copy_client.cc:791] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:40.816664 1846 tablet_copy_client.cc:655] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:40.821610 1846 tablet_copy_client.cc:531] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:40.827782 1841 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:40.828805 1841 tablet_bootstrap.cc:492] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:40.831212 1841 ts_tablet_manager.cc:1397] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.005s
I20260806 04:00:40.835027 1841 raft_consensus.cc:348] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.839946 1841 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:40.843422 1846 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:40.841945 1841 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.847589 1842 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:40.848873 1842 tablet_bootstrap.cc:492] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:40.849598 1842 ts_tablet_manager.cc:1397] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.014s
I20260806 04:00:40.855002 1841 ts_tablet_manager.cc:1428] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.023s user 0.007s sys 0.001s
I20260806 04:00:40.857285 1842 raft_consensus.cc:348] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.858212 1842 raft_consensus.cc:729] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:40.858830 1842 consensus_queue.cc:260] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.861907 1842 ts_tablet_manager.cc:1428] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 04:00:40.863972 32505 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-187160a8bf734a23b90fcc49bc24b10b received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:40.864377 32505 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-187160a8bf734a23b90fcc49bc24b10b on tablet 187160a8bf734a23b90fcc49bc24b10b with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:40.864880 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-f9fdfcc3345c4c86a7d974030c67c271 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:40.865286 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-f9fdfcc3345c4c86a7d974030c67c271 on tablet f9fdfcc3345c4c86a7d974030c67c271 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:40.871040 1846 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:40.871757 1846 tablet_bootstrap.cc:492] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:40.872435 1846 ts_tablet_manager.cc:1397] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.009s
I20260806 04:00:40.874897 1846 raft_consensus.cc:348] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.875659 1846 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:40.876183 1846 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:40.878440 1846 ts_tablet_manager.cc:1428] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:00:40.880292 32505 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-1e2f839ac9f94c2f980cc895f641607f received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:40.880667 32505 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-1e2f839ac9f94c2f980cc895f641607f on tablet 1e2f839ac9f94c2f980cc895f641607f with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:41.142537 1189 raft_consensus.cc:1203] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:41.165087 1189 raft_consensus.cc:1203] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:41.177027 1189 raft_consensus.cc:1203] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:41.283564 1188 raft_consensus.cc:1203] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:41.559782 1860 raft_consensus.cc:1050] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:41.562680 1860 consensus_queue.cc:237] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:41.571779 32749 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.572927 32466 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.574002 1859 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:41.575793 1729 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:41.576190 1188 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.583547 1860 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:41.593269 1777 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.604264 1188 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.604074 32466 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.617386 32743 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.621651 32326 catalog_manager.cc:5495] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.643821 1729 raft_consensus.cc:1050] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:41.646123 1729 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:41.656386 1188 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.657555 1777 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:41.660492 32743 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.663266 32469 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.661734 1729 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260806 04:00:41.668762 1859 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:41.672514 32615 consensus_queue.cc:237] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:41.680171 32749 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:41.681497 1729 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:41.684480 1188 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:41.683912 1777 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.689728 1859 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.690948 32743 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.702564 32469 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.710039 1886 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:41.718816 32749 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.718902 32324 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.733916 1192 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.735046 1189 raft_consensus.cc:2932] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.744040 32322 catalog_manager.cc:5495] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:41.747920 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 305cbd07ce124048b7ac2f99b2aa40db with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:41.747802 1657 raft_consensus.cc:1050] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:41.750541 1657 consensus_queue.cc:237] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:41.757041 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet 305cbd07ce124048b7ac2f99b2aa40db with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:41.762550 32604 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:41.771996 32467 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:41.774889 1892 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:41.774906 1192 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:41.778350 1889 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:41.784781 1889 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.785719 32466 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.791810 1189 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.792596 32749 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.793573 1714 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:41.794436 1822 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:41.799467 1895 tablet_replica.cc:330] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:41.802403 32615 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.802069 1189 raft_consensus.cc:2932] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.809242 1822 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 04:00:41.811165 1895 raft_consensus.cc:2229] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:41.817685 1895 raft_consensus.cc:2258] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:41.812651 32322 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.826959 1895 ts_tablet_manager.cc:1905] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:41.831704 1657 raft_consensus.cc:1050] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:41.834142 1657 consensus_queue.cc:237] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:41.857632 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f9fdfcc3345c4c86a7d974030c67c271 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:41.870740 1899 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.879395 32615 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.893801 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet f9fdfcc3345c4c86a7d974030c67c271 with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:41.895804 1192 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.896723 1895 ts_tablet_manager.cc:1918] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:41.901487 1895 log.cc:1196] T 305cbd07ce124048b7ac2f99b2aa40db P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/305cbd07ce124048b7ac2f99b2aa40db
I20260806 04:00:41.903678 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet 305cbd07ce124048b7ac2f99b2aa40db (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:41.913403 32322 catalog_manager.cc:5495] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.930778 1907 tablet_replica.cc:330] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:41.931234 32743 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.935108 1907 raft_consensus.cc:2229] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:41.922853 32749 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:41.936235 1907 raft_consensus.cc:2258] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:41.941817 1907 ts_tablet_manager.cc:1905] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:41.942142 1192 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.941782 1822 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:41.945618 1714 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:41.948782 32604 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:41.953938 1657 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:42.007225 32466 consensus_queue.cc:237] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:42.018893 32743 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:42.021903 1190 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:42.024549 1822 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:42.025645 1907 ts_tablet_manager.cc:1918] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:42.025926 1657 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:42.036785 1907 log.cc:1196] T f9fdfcc3345c4c86a7d974030c67c271 P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/f9fdfcc3345c4c86a7d974030c67c271
I20260806 04:00:42.034833 1896 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.044521 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet f9fdfcc3345c4c86a7d974030c67c271 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:42.037135 32749 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.059217 1822 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.063654 32741 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.083745 32324 catalog_manager.cc:5495] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:42.087381 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 58964.6us
I20260806 04:00:42.092880 1190 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.106024 32324 catalog_manager.cc:5495] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.109120 32604 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.119277 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 187160a8bf734a23b90fcc49bc24b10b with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:42.130389 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 187160a8bf734a23b90fcc49bc24b10b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:42.141574 1192 raft_consensus.cc:2932] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.154472 1918 tablet_replica.cc:330] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:42.156155 1918 raft_consensus.cc:2229] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:42.158015 1918 raft_consensus.cc:2258] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:42.162624 1918 ts_tablet_manager.cc:1905] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:42.181648 32466 consensus_queue.cc:237] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:42.186546 1918 ts_tablet_manager.cc:1918] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:42.187856 1190 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:42.187111 1918 log.cc:1196] T 187160a8bf734a23b90fcc49bc24b10b P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/187160a8bf734a23b90fcc49bc24b10b
I20260806 04:00:42.188124 32604 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:42.190030 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 187160a8bf734a23b90fcc49bc24b10b (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:42.190117 1896 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:42.191412 1917 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:42.200140 1896 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.204774 1190 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.206079 32604 raft_consensus.cc:2932] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:42.214246 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1e2f839ac9f94c2f980cc895f641607f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:42.222827 32324 catalog_manager.cc:5495] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:42.243265 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2f839ac9f94c2f980cc895f641607f with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:42.244263 1907 tablet_replica.cc:330] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:42.245707 1907 raft_consensus.cc:2229] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:42.246562 1907 raft_consensus.cc:2258] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:42.250293 1907 ts_tablet_manager.cc:1905] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:42.266655 1907 ts_tablet_manager.cc:1918] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:42.267055 1907 log.cc:1196] T 1e2f839ac9f94c2f980cc895f641607f P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/1e2f839ac9f94c2f980cc895f641607f
I20260806 04:00:42.268724 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 1e2f839ac9f94c2f980cc895f641607f (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:43.181435 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:43.251621 32615 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:43.252367 32615 raft_consensus.cc:595] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:43.274997 32469 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:43.275556 32469 raft_consensus.cc:595] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:43.291770 32741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:43.292323 32741 raft_consensus.cc:595] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:43.294739 32469 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:43.295300 32469 raft_consensus.cc:595] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:00:43.297734 1919 raft_consensus.cc:979] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:43.297713 32469 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:43.298354 32469 raft_consensus.cc:595] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:43.298354 1651 raft_consensus.cc:1067] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:43.299983 1190 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37300
I20260806 04:00:43.300554 1190 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:43.300868 1190 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:43.304981 1920 raft_consensus.cc:979] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:43.305438 1920 raft_consensus.cc:1067] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:43.307703 32615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41308
I20260806 04:00:43.308218 32615 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:43.308563 32615 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.308341 32477 consensus_queue.cc:237] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:43.310433 1190 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.312202 1190 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:43.313200 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:43.313735 32467 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:43.314292 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:43.314857 32741 raft_consensus.cc:3032] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:00:43.315171 32741 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:43.315889 32741 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.316262 32469 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:43.316275 32615 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.317062 32469 raft_consensus.cc:595] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:43.317124 32741 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:43.319291 32615 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:43.319599 32467 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:00:43.320437 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:43.321007 32749 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.324132 32741 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:00:43.327564 32741 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:43.324036 1137 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:00:43.328653 1896 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:43.329401 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:43.330195 32467 raft_consensus.cc:3032] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:43.331593 32467 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:43.332161 32615 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:43.331740 1936 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 04:00:43.333931 32427 consensus_peers.cc:487] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd55a7d68ee84cb7a69c962f984f9372. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:43.333196 32467 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.334388 32749 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:43.335695 1896 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:43.336602 32467 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.333789 32477 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:43.338084 32477 raft_consensus.cc:595] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:43.339224 32553 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:43.340413 1922 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:43.340853 1922 raft_consensus.cc:686] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:43.341645 1922 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.342523 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260806 04:00:43.344461 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:43.344888 32467 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:43.345222 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:43.355603 1936 raft_consensus.cc:686] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:43.356755 1936 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.358080 1899 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.359673 32604 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.371209 32324 catalog_manager.cc:5495] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:43.377095 32749 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:43.375401 32321 catalog_manager.cc:5495] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:43.384753 32322 catalog_manager.cc:5495] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 3 to 4, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 4 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:43.384799 32741 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.407356 32469 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:43.409400 1955 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:43.411412 32604 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:43.411818 32682 consensus_peers.cc:487] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7172fff185b4788acb609d2314e4b4b. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:43.415325 1922 raft_consensus.cc:979] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:43.415928 1922 raft_consensus.cc:1067] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:43.420188 1190 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37314
I20260806 04:00:43.420786 1190 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:43.421144 1190 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.423225 1914 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:43.433027 1190 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.438069 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:43.438727 32467 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.447019 32467 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:43.450428 1137 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:00:43.451412 1936 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:43.451897 1936 raft_consensus.cc:686] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:43.452785 1936 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.461170 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:43.461757 32615 raft_consensus.cc:3032] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:43.462029 32615 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:43.462702 32615 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.463769 32615 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.470983 32615 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:43.472414 1190 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:43.526108 1651 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.537606 1899 raft_consensus.cc:979] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:43.538882 1899 raft_consensus.cc:1067] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:43.588814 32469 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.578773 32324 catalog_manager.cc:5495] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:43.604656 32326 catalog_manager.cc:5495] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:43.613853 1190 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37290
I20260806 04:00:43.614553 1190 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:43.614903 1190 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.629747 1896 raft_consensus.cc:979] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:43.630303 1896 raft_consensus.cc:1067] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:43.631872 32614 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41308
I20260806 04:00:43.632531 32614 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:43.633113 32614 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.640749 1190 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.642629 32614 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.644510 1190 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:43.645677 32614 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:00:43.634330 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 103013us
I20260806 04:00:43.645778 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:43.646373 32615 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.651464 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:43.652031 32743 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.652395 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:43.652928 32467 raft_consensus.cc:3032] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:43.653182 32467 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:43.653848 32467 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.654803 32467 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.658514 32615 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:43.659547 1136 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:43.660384 1936 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:43.660851 1936 raft_consensus.cc:686] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:43.660712 32604 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.655946 32749 consensus_queue.cc:237] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:43.661727 1936 consensus_queue.cc:237] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.669889 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:43.670493 32469 raft_consensus.cc:3032] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:43.670706 32469 raft_consensus.cc:729] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:43.671331 32469 consensus_queue.cc:260] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:43.672501 32469 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.662233 32743 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:43.676214 1920 raft_consensus.cc:979] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:43.676718 1920 raft_consensus.cc:1067] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
W20260806 04:00:43.679728 32682 consensus_peers.cc:487] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a12b9e678e6049be86bca4e981f90379. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:43.681440 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:43.681595 32467 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:43.682318 32743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:00:43.682900 32743 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:43.683305 32743 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.684854 1966 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:43.685357 1966 raft_consensus.cc:686] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:43.686185 1966 consensus_queue.cc:237] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.690210 32322 catalog_manager.cc:5495] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:43.693629 32604 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:43.694988 32467 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:43.696637 1979 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:43.698387 1956 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:43.699889 32743 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.701740 32743 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:43.702359 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:43.702947 32466 raft_consensus.cc:3032] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:43.703245 32466 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:43.703917 32466 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.704399 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:43.704949 32466 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.705003 32614 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.686146 32469 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:43.713858 32614 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:00:43.714454 32466 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:00:43.714818 32682 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:43.715770 1956 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:43.716389 1956 raft_consensus.cc:686] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:43.713346 1971 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.717255 1956 consensus_queue.cc:237] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.719506 32466 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.726490 32604 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.725517 32322 catalog_manager.cc:5495] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:43.732901 32467 consensus_queue.cc:237] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:00:43.747109 32427 consensus_peers.cc:487] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2ad471021a947e9a45d57c7be704453. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:43.747526 32743 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:43.748762 32604 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:43.749413 1981 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:43.750349 1896 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:43.752491 32326 catalog_manager.cc:5495] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.762542 32324 catalog_manager.cc:5495] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:43.767014 32749 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:43.768543 1967 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:43.771523 1896 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.779093 32614 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.800376 1920 raft_consensus.cc:979] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:43.800891 1920 raft_consensus.cc:1067] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:43.804543 32614 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41308
I20260806 04:00:43.810817 32614 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:43.815635 32614 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.814379 32324 catalog_manager.cc:5495] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.829584 32467 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:43.831576 1967 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:43.872180 32614 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.874729 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:43.875327 32467 raft_consensus.cc:3032] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:43.875612 32467 raft_consensus.cc:729] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:43.876299 32467 consensus_queue.cc:260] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.877338 32467 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.878294 1998 ts_tablet_manager.cc:927] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:43.884267 32614 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:43.893046 32467 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:43.894955 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:43.898507 1922 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:43.899031 1922 raft_consensus.cc:686] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:43.902217 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:43.902879 32749 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:43.899899 1922 consensus_queue.cc:237] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:43.907192 32469 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:00:43.908638 1936 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:00:43.914338 32749 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:43.938292 1997 ts_tablet_manager.cc:927] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:43.942298 1998 tablet_copy_client.cc:316] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:43.944590 32743 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:43.948350 32505 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet bd55a7d68ee84cb7a69c962f984f9372 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:43.948973 32505 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet bd55a7d68ee84cb7a69c962f984f9372 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-bd55a7d68ee84cb7a69c962f984f9372
I20260806 04:00:43.970166 32749 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:00:43.977025 32505 tablet_copy_source_session.cc:205] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:43.975152 32322 catalog_manager.cc:5495] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:43.979173 1938 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:00:43.982869 1997 tablet_copy_client.cc:316] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:43.985999 1998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd55a7d68ee84cb7a69c962f984f9372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:43.986708 300 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet d7172fff185b4788acb609d2314e4b4b from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:43.987259 300 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet d7172fff185b4788acb609d2314e4b4b from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-d7172fff185b4788acb609d2314e4b4b
I20260806 04:00:44.000905 300 tablet_copy_source_session.cc:205] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:44.014542 32469 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:44.016836 2016 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:44.018919 1997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7172fff185b4788acb609d2314e4b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:44.130627 32604 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:44.133670 1998 tablet_copy_client.cc:791] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:44.134382 1998 tablet_copy_client.cc:655] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:44.158612 32604 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:44.155742 2022 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
W20260806 04:00:44.156620 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 52021.1us
I20260806 04:00:44.164565 32469 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:44.173770 1997 tablet_copy_client.cc:791] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:44.174475 1997 tablet_copy_client.cc:655] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:44.179155 2016 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:44.182951 32469 raft_consensus.cc:1261] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:44.188004 1998 tablet_copy_client.cc:531] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:44.189581 2032 ts_tablet_manager.cc:927] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:44.190151 2029 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:44.197778 2028 consensus_queue.cc:1035] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:44.199045 1998 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:44.213539 32743 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:44.230151 32478 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:44.235726 1859 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:44.248401 1922 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 04:00:44.289178 1137 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletCopyService.FetchData -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 101525us
I20260806 04:00:44.292945 1997 tablet_copy_client.cc:531] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:44.296657 32615 raft_consensus.cc:1261] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:44.298426 32749 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:44.298372 1956 consensus_queue.cc:1035] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:44.312484 32469 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:44.314276 2025 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:44.316602 1966 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:44.518608 1997 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:44.522372 2040 ts_tablet_manager.cc:927] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:44.541836 2032 tablet_copy_client.cc:316] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:44.549541 300 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet a12b9e678e6049be86bca4e981f90379 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:44.554828 300 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet a12b9e678e6049be86bca4e981f90379 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-a12b9e678e6049be86bca4e981f90379
I20260806 04:00:44.611577 2040 tablet_copy_client.cc:316] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:44.694590 300 tablet_copy_source_session.cc:205] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:44.750957 32505 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet d2ad471021a947e9a45d57c7be704453 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:44.751631 32505 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet d2ad471021a947e9a45d57c7be704453 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-d2ad471021a947e9a45d57c7be704453
I20260806 04:00:44.802135 2032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12b9e678e6049be86bca4e981f90379. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 04:00:44.808965 2036 log.cc:924] Time spent T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.252s user 0.007s sys 0.000s
W20260806 04:00:44.888839 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 244748us
I20260806 04:00:44.938768 1997 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:44.940238 1997 tablet_bootstrap.cc:492] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:44.941223 1997 ts_tablet_manager.cc:1397] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.423s user 0.028s sys 0.019s
I20260806 04:00:44.962541 1997 raft_consensus.cc:348] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:44.963531 1997 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:44.964330 1997 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:44.967262 1997 ts_tablet_manager.cc:1428] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260806 04:00:44.972014 1998 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:44.973224 1998 tablet_bootstrap.cc:492] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:44.974229 1998 ts_tablet_manager.cc:1397] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.775s user 0.028s sys 0.044s
I20260806 04:00:45.110343 2032 tablet_copy_client.cc:791] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:45.111129 2032 tablet_copy_client.cc:655] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:44.604997 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:45.230250 32505 tablet_copy_source_session.cc:205] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
W20260806 04:00:45.234503 2035 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.711s user 0.002s sys 0.003s
I20260806 04:00:45.236131 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1067) took 1031 ms. Trace:
I20260806 04:00:45.236649 32466 rpcz_store.cc:276] 0806 04:00:44.204768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:44.204924 (+ 156us) service_pool.cc:227] Handling call
0806 04:00:44.205225 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:00:44.205514 (+ 289us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:00:44.205538 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:00:44.205923 (+ 385us) log.cc:842] Serialized 26 byte log entry
0806 04:00:44.533689 (+327766us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:00:44.533729 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:00:44.533797 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:00:45.234230 (+700433us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b240019b9c8. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a184cf4 00007fd16a182dfb 00007fd16a211559 00007fd16a2111e5 00007fd16a21082d 00007fd16cebf46e 00007fd16ced6c0c 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551
0806 04:00:45.235706 (+ 1476us) raft_consensus.cc:1671] finished
0806 04:00:45.235728 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:00:45.236000 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3142,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":88,"spinlock_wait_cycles":19902336,"thread_start_us":327427,"threads_started":1,"wal-append.queue_time_us":317356}
W20260806 04:00:45.276681 32427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.061s sys 0.133s
W20260806 04:00:45.277084 32427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.745s user 0.061s sys 0.133s
I20260806 04:00:45.278779 300 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-d7172fff185b4788acb609d2314e4b4b received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:45.279392 300 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-d7172fff185b4788acb609d2314e4b4b on tablet d7172fff185b4788acb609d2314e4b4b with peer d4a238f055a44f459c62c224daa4d541
W20260806 04:00:45.282882 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.011s sys 0.023s
W20260806 04:00:45.283247 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.011s sys 0.023s
W20260806 04:00:45.283496 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 658312us
I20260806 04:00:45.309521 1185 raft_consensus.cc:1203] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:45.355439 2032 tablet_copy_client.cc:531] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260806 04:00:45.369016 32430 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.007s sys 0.028s
W20260806 04:00:45.369417 32430 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.762s user 0.008s sys 0.028s
I20260806 04:00:45.382836 1998 raft_consensus.cc:348] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:00:45.394419 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.006s sys 0.011s
W20260806 04:00:45.394827 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.795s user 0.009s sys 0.011s
W20260806 04:00:45.396893 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 821216us
I20260806 04:00:45.405958 1998 raft_consensus.cc:729] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:45.408595 1998 consensus_queue.cc:260] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:45.423609 2040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ad471021a947e9a45d57c7be704453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:45.441485 32478 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:45.457391 1998 ts_tablet_manager.cc:1428] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.480s user 0.010s sys 0.006s
I20260806 04:00:45.467845 32478 raft_consensus.cc:595] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:45.480325 32505 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-bd55a7d68ee84cb7a69c962f984f9372 received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:45.480763 32505 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-bd55a7d68ee84cb7a69c962f984f9372 on tablet bd55a7d68ee84cb7a69c962f984f9372 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:45.487608 2032 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:45.488225 32474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:45.499428 32474 raft_consensus.cc:595] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:45.523134 32467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:45.523874 32467 raft_consensus.cc:595] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:45.527403 32603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:45.527995 32603 raft_consensus.cc:595] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:45.530704 32743 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:45.531227 32743 raft_consensus.cc:595] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:45.533150 2040 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37290 (request call id 290) took 1325 ms. Trace:
I20260806 04:00:45.535087 32603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:45.540180 32603 raft_consensus.cc:595] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:45.541188 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 04:00:45.539635 2040 rpcz_store.cc:276] 0806 04:00:44.207963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:44.209511 (+ 1548us) service_pool.cc:227] Handling call
0806 04:00:44.522871 (+313360us) ts_tablet_manager.cc:928] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
0806 04:00:45.418913 (+896042us) tablet_copy_client.cc:361] Tablet copy session begun
0806 04:00:45.483333 (+ 64420us) tablet_metadata.cc:641] Metadata flushed
0806 04:00:45.490849 (+ 7516us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 04:00:45.532416 (+ 41567us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 04:00:45.532706 (+ 290us) tablet_replica.cc:205] Creating consensus instance
0806 04:00:45.533013 (+ 307us) ts_tablet_manager.cc:983] Replica registered.
0806 04:00:45.533073 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":18007,"tablet-copy.queue_time_us":198108,"thread_start_us":141663,"threads_started":2}
I20260806 04:00:45.542979 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:45.544366 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:45.548681 2032 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:45.549700 2032 tablet_bootstrap.cc:492] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:45.550576 2032 ts_tablet_manager.cc:1397] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.063s user 0.021s sys 0.001s
I20260806 04:00:45.550616 2040 tablet_copy_client.cc:791] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:45.552809 2040 tablet_copy_client.cc:655] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:45.554775 2032 raft_consensus.cc:348] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:45.556046 2032 raft_consensus.cc:729] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:45.556692 2032 consensus_queue.cc:260] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:45.559000 2032 ts_tablet_manager.cc:1428] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260806 04:00:45.560707 32766 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-a12b9e678e6049be86bca4e981f90379 received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:45.561255 32766 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-a12b9e678e6049be86bca4e981f90379 on tablet a12b9e678e6049be86bca4e981f90379 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:45.564845 2040 tablet_copy_client.cc:531] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:45.593122 2040 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:45.628857 2040 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:45.630122 2040 tablet_bootstrap.cc:492] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:45.630932 2040 ts_tablet_manager.cc:1397] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.038s user 0.021s sys 0.012s
I20260806 04:00:45.635576 2040 raft_consensus.cc:348] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:45.636863 2040 raft_consensus.cc:729] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:45.637885 2040 consensus_queue.cc:260] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:45.641456 2040 ts_tablet_manager.cc:1428] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.010s user 0.004s sys 0.008s
I20260806 04:00:45.652797 2023 raft_consensus.cc:1050] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
W20260806 04:00:45.653383 2023 raft_consensus.cc:1054] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Unable to promote non-voter d4a238f055a44f459c62c224daa4d541: Service unavailable: leader transfer in progress
I20260806 04:00:45.698553 32505 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-d2ad471021a947e9a45d57c7be704453 received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:45.699029 32505 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-d2ad471021a947e9a45d57c7be704453 on tablet d2ad471021a947e9a45d57c7be704453 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:45.732131 2064 raft_consensus.cc:979] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:45.732868 1966 raft_consensus.cc:1067] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:45.735378 32471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:00:45.736123 32471 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:45.751070 2057 raft_consensus.cc:979] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:45.736651 32471 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:45.763877 2049 raft_consensus.cc:1067] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:45.778633 32471 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:45.780856 32471 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:45.802235 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:45.802990 32604 raft_consensus.cc:3032] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:45.803303 32604 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:45.804205 32604 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:45.836892 32604 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:45.862805 32604 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:45.867712 32427 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:45.881842 1899 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:45.882551 1899 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:45.883849 1899 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:45.905136 2072 raft_consensus.cc:979] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:45.905710 2072 raft_consensus.cc:1067] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:45.907835 32474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:00:45.908699 32474 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:45.909180 32474 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:00:45.934111 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 66482.4us
I20260806 04:00:45.949358 32474 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:45.952421 32474 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:45.956281 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:45.957635 32749 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:45.969930 32749 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:45.991550 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:45.992213 32749 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:45.988947 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:45.999709 32749 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:46.023136 32603 raft_consensus.cc:3032] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:46.038141 1189 raft_consensus.cc:1203] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:46.034557 32603 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:46.039155 32603 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:46.040501 32603 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.045801 1187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37290
W20260806 04:00:46.046047 32369 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 30649
I20260806 04:00:46.046479 1187 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:46.046849 1187 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:00:46.047113 32369 debug-util.cc:398] Leaking SignalData structure 0x7b080008e720 after lost signal to thread 32373
W20260806 04:00:46.048318 32369 debug-util.cc:398] Leaking SignalData structure 0x7b08001b3a00 after lost signal to thread 32530
W20260806 04:00:46.052372 32369 debug-util.cc:398] Leaking SignalData structure 0x7b08000bd980 after lost signal to thread 32659
I20260806 04:00:46.056294 1186 raft_consensus.cc:1203] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:46.058296 32428 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:00:46.056224 32369 debug-util.cc:398] Leaking SignalData structure 0x7b080030a6a0 after lost signal to thread 326
W20260806 04:00:46.060385 32369 debug-util.cc:398] Leaking SignalData structure 0x7b08002ea2c0 after lost signal to thread 1231
I20260806 04:00:46.067426 2085 raft_consensus.cc:1050] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
W20260806 04:00:46.067883 2085 raft_consensus.cc:1054] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Unable to promote non-voter d4a238f055a44f459c62c224daa4d541: Service unavailable: leader transfer in progress
I20260806 04:00:46.065052 32603 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
W20260806 04:00:46.069600 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 80183.6us
I20260806 04:00:46.072300 2097 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:46.074391 1920 raft_consensus.cc:979] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:46.074738 2097 raft_consensus.cc:686] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:46.075201 1920 raft_consensus.cc:1067] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:46.075479 1189 raft_consensus.cc:1203] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:46.078208 2085 raft_consensus.cc:979] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:46.090809 1979 raft_consensus.cc:1067] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:46.092343 1185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37290
I20260806 04:00:46.093076 1185 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:46.093446 1185 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.076278 2097 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:46.096107 1920 raft_consensus.cc:979] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:46.097110 2054 raft_consensus.cc:1067] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:46.097383 32471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59514
I20260806 04:00:46.098155 32471 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:46.098508 32471 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.136353 1189 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37290
I20260806 04:00:46.078343 1187 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.136991 1189 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:46.137426 1189 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:46.141889 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:46.142748 32474 raft_consensus.cc:3032] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:46.143088 32474 raft_consensus.cc:729] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:46.144354 32474 consensus_queue.cc:260] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.145787 32474 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.148414 1187 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:46.154490 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:46.155211 32609 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.164299 32609 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:46.168550 1136 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:46.170934 1189 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.173091 2100 raft_consensus.cc:2781] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:46.174084 2100 raft_consensus.cc:686] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:46.175388 2100 consensus_queue.cc:237] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.180167 1185 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.183018 1185 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:46.184631 1187 raft_consensus.cc:1226] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:00:46.187451 32471 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:46.189138 1189 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:46.189538 32474 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:46.190400 32471 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:46.184929 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:46.193933 32749 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.196957 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:46.197644 32748 raft_consensus.cc:3032] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:46.197939 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:46.198551 32741 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:46.197906 32748 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:46.201161 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:46.200920 32748 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:46.202852 32748 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.206262 32741 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:00:46.201766 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:46.207512 32478 raft_consensus.cc:3032] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:00:46.207916 32478 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:46.210081 32478 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.211375 32478 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:46.197971 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:46.212746 32470 raft_consensus.cc:3032] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:46.213186 32470 raft_consensus.cc:729] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:46.201922 32603 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.217499 32749 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:46.215533 32470 consensus_queue.cc:260] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.239034 32470 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:46.226895 32478 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:00:46.236908 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:46.251555 32322 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:46.253501 32322 catalog_manager.cc:5495] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:46.254145 32470 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:46.257701 1938 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:00:46.258356 1938 raft_consensus.cc:686] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:46.259532 1938 consensus_queue.cc:237] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.262106 32748 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:46.272892 32428 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:46.273849 2054 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:46.274595 2054 raft_consensus.cc:686] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:46.275053 32603 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:46.273829 1137 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:00:46.279100 2100 raft_consensus.cc:2781] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:46.282392 2102 raft_consensus.cc:1050] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:46.288578 32324 catalog_manager.cc:5495] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:46.291510 2100 raft_consensus.cc:686] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:46.275832 2054 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:46.292644 2100 consensus_queue.cc:237] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.303146 32741 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:46.294919 2102 consensus_queue.cc:237] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.314106 2069 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260806 04:00:46.402027 32609 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:46.415063 1899 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260806 04:00:46.437638 32741 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.439149 2054 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:46.461295 1185 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.464648 1920 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:46.470171 32609 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.487706 1920 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260806 04:00:46.531903 2075 raft_consensus.cc:1050] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:46.545734 2075 consensus_queue.cc:237] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.554245 32609 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:46.555531 1970 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:46.556173 1920 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:46.561355 32614 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:46.566397 1185 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:46.574445 32745 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:46.583570 32741 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.585958 1970 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:46.587252 1185 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.588690 1970 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:46.594290 32614 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.597074 1970 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:46.600678 32478 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:46.603395 2100 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:46.618744 2053 raft_consensus.cc:1050] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:46.622278 2053 consensus_queue.cc:237] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:46.644865 1920 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:46.648433 32746 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:46.665498 32603 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:46.666814 32471 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:00:46.669649 2016 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:46.679730 2016 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:46.688069 32603 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:46.691211 2065 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:46.697055 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:46.703593 32324 catalog_manager.cc:5495] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 3 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:46.709183 32324 catalog_manager.cc:5495] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:46.710954 32324 catalog_manager.cc:5495] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:46.723927 32741 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:46.729902 2054 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:46.750767 32741 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:46.752662 2016 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:46.745434 1187 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:46.772516 32604 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.772569 1189 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.775398 2026 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:46.777810 2026 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 04:00:46.778250 32652 debug-util.cc:398] Leaking SignalData structure 0x7b080016f4e0 after lost signal to thread 30649
W20260806 04:00:46.779381 32652 debug-util.cc:398] Leaking SignalData structure 0x7b080016fbc0 after lost signal to thread 32373
W20260806 04:00:46.780867 32652 debug-util.cc:398] Leaking SignalData structure 0x7b080011ee80 after lost signal to thread 32530
I20260806 04:00:46.780927 32321 catalog_manager.cc:5495] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 04:00:46.783625 32652 debug-util.cc:398] Leaking SignalData structure 0x7b08003612e0 after lost signal to thread 32659
W20260806 04:00:46.784897 32652 debug-util.cc:398] Leaking SignalData structure 0x7b080004d580 after lost signal to thread 326
W20260806 04:00:46.786643 32652 debug-util.cc:398] Leaking SignalData structure 0x7b08001e45a0 after lost signal to thread 1231
I20260806 04:00:46.810225 32321 catalog_manager.cc:5495] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:46.828624 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 74620.5us
I20260806 04:00:46.831641 32614 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:46.921011 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 60620.5us
I20260806 04:00:46.927050 32478 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:46.927726 32478 raft_consensus.cc:595] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:46.934587 1185 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Advancing to term 2
I20260806 04:00:46.938284 32748 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:00:46.971982 32473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:46.972817 32473 raft_consensus.cc:595] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:46.973251 32473 raft_consensus.cc:603] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:00:46.934712 32470 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:46.986497 1938 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:46.970625 1185 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:47.014587 32475 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:47.033448 32474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:47.034251 2057 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:47.034996 32474 raft_consensus.cc:595] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:47.035244 1938 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:47.017582 32473 consensus_queue.cc:237] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.038691 2107 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:47.060681 32741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:47.061321 32741 raft_consensus.cc:595] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:47.063701 1185 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260806 04:00:47.070775 2165 log.cc:924] Time spent T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260806 04:00:47.099786 32741 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:47.116163 32473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57c0e905630e43668379f2f95684775f"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:47.116770 32473 raft_consensus.cc:595] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:47.119421 2097 raft_consensus.cc:979] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:47.120143 2097 raft_consensus.cc:1067] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:47.085951 1970 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260806 04:00:47.124657 2102 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:47.134984 32746 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:47.114902 1979 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.149412 1186 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37290
I20260806 04:00:47.164987 1186 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:47.167263 1186 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:00:47.229444 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 88770.6us
W20260806 04:00:47.245115 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 65205.1us
I20260806 04:00:47.264163 32603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:47.265018 32603 raft_consensus.cc:595] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:47.155025 1187 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:47.276896 2190 log.cc:924] Time spent T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
W20260806 04:00:47.277966 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 102762us
I20260806 04:00:47.239154 2185 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20260806 04:00:47.302548 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 04:00:47.304418 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:47.305148 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:47.369473 32475 consensus_queue.cc:237] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:00:47.374854 2163 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.445s user 0.003s sys 0.003s
I20260806 04:00:47.388571 1189 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:47.412870 32609 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:47.425946 2180 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.429049 1189 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.409092 2065 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260806 04:00:47.447439 2075 raft_consensus.cc:1050] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:47.449249 2054 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:47.452527 2186 raft_consensus.cc:979] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
W20260806 04:00:47.457567 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 227106us
I20260806 04:00:47.459339 2186 raft_consensus.cc:1067] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:47.468653 1920 raft_consensus.cc:979] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:47.469172 1920 raft_consensus.cc:1067] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:47.474076 32609 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.481071 1192 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57c0e905630e43668379f2f95684775f"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37290
I20260806 04:00:47.481667 1192 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:47.481956 1192 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.474442 2075 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.485565 2054 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.504788 1185 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.507967 32471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59514
I20260806 04:00:47.508757 32471 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:47.509192 32471 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.514228 32746 raft_consensus.cc:2932] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.521004 32603 raft_consensus.cc:2932] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.517014 1186 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.533521 1186 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:47.536486 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:47.536819 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:47.541270 32604 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.541394 32475 raft_consensus.cc:3032] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:47.542311 1192 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.542721 32475 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:47.543478 32475 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.546667 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:47.547107 32475 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.558202 2072 raft_consensus.cc:979] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:47.558825 2072 raft_consensus.cc:1067] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:47.545691 1192 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:47.568100 1187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37314
I20260806 04:00:47.598013 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d2ad471021a947e9a45d57c7be704453 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:47.581027 32610 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:47.582974 1189 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:47.586717 32745 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:47.593971 32604 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:47.547283 32614 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.570314 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:47.604903 32474 raft_consensus.cc:3032] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:47.605275 32474 raft_consensus.cc:729] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:47.606003 32474 consensus_queue.cc:260] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.607199 32474 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.597434 1187 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:47.610167 1136 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:47.613245 2057 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260806 04:00:47.613550 1187 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.640799 2175 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:47.641507 2175 raft_consensus.cc:686] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:47.642498 2175 consensus_queue.cc:237] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.628924 2176 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260806 04:00:47.648957 1920 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:47.651046 2065 raft_consensus.cc:979] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:47.651564 2065 raft_consensus.cc:1067] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:47.655808 32609 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41308
I20260806 04:00:47.656358 32609 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:47.656735 32609 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:47.666033 32470 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.691504 32474 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:47.694300 32471 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:47.697903 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet bd55a7d68ee84cb7a69c962f984f9372 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:47.700716 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:47.701274 32746 raft_consensus.cc:3032] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:47.701529 32746 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:47.702180 32746 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:47.700193 32471 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:47.703001 32475 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:47.713446 32475 raft_consensus.cc:1226] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:00:47.714430 32614 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:47.716334 32615 raft_consensus.cc:1226] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:00:47.704226 1920 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.703032 32746 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.709957 32609 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:47.696255 32321 catalog_manager.cc:5495] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.717073 32604 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.725929 1137 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:00:47.725871 32609 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:47.727535 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:47.728197 32474 raft_consensus.cc:3032] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:00:47.778447 32474 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:47.779362 32474 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:47.780543 32474 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:47.787650 2174 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:47.791299 32474 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:00:47.724818 1187 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.791379 2174 raft_consensus.cc:686] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:47.795207 32553 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:47.722091 32321 catalog_manager.cc:5495] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.786972 32746 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:47.798009 32428 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:47.808828 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:47.809546 32746 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:47.792873 1186 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.818107 32746 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:00:47.813663 2075 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:47.820261 2210 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:00:47.820513 2075 raft_consensus.cc:686] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:47.821424 2075 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:47.810282 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:47.836211 2210 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:47.837188 2210 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:47.793123 2174 consensus_queue.cc:237] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.838392 32603 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.857549 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:47.858184 32609 raft_consensus.cc:3032] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:47.858482 32609 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:47.859197 32609 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.861441 32609 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.890802 32603 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:47.892864 32609 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:47.893832 1187 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:47.910137 1136 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:47.910712 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:47.914570 32746 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:47.915925 2100 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:47.916538 2100 raft_consensus.cc:686] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:47.917469 2100 consensus_queue.cc:237] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:47.946710 32745 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:47.966499 32746 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:48.065341 32320 catalog_manager.cc:5495] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:48.077445 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet d2ad471021a947e9a45d57c7be704453 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:48.089046 2231 tablet_replica.cc:330] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:48.165030 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet bd55a7d68ee84cb7a69c962f984f9372 with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:48.181715 32321 catalog_manager.cc:5495] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:48.197209 32473 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:48.196947 32748 consensus_queue.cc:237] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:48.198894 2175 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:48.190343 32321 catalog_manager.cc:5495] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:48.217607 1186 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:48.221401 32603 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:48.223486 2201 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:48.225781 2201 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:48.253520 2232 tablet_replica.cc:330] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:48.255162 2232 raft_consensus.cc:2229] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:48.261162 2207 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:48.263985 2231 raft_consensus.cc:2229] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:48.271230 2231 raft_consensus.cc:2258] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:48.274099 32604 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:48.275800 2175 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:48.279093 2231 ts_tablet_manager.cc:1905] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:48.285655 1187 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:48.282637 32320 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 2 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 3 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:48.281188 32603 raft_consensus.cc:2932] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:48.306622 32614 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:48.316603 2175 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:48.318238 2232 raft_consensus.cc:2258] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:48.322968 32467 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:00:48.340698 32322 catalog_manager.cc:5495] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:48.342077 2232 ts_tablet_manager.cc:1905] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:48.344614 32475 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:48.349278 2027 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:48.354245 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a12b9e678e6049be86bca4e981f90379 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:48.355531 32745 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:48.342373 32322 catalog_manager.cc:5495] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:48.357821 2175 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:48.358459 32322 catalog_manager.cc:5495] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:48.388465 32748 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:00:48.389425 32746 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:48.390848 1186 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:48.393244 2068 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:48.397737 2054 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:48.399747 2098 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:48.430684 32609 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:48.429072 2231 ts_tablet_manager.cc:1918] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:48.432021 2231 log.cc:1196] T d2ad471021a947e9a45d57c7be704453 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/d2ad471021a947e9a45d57c7be704453
I20260806 04:00:48.448995 32474 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:48.475391 2174 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:48.481745 32604 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:48.491006 2102 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:48.443073 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:48.542151 2174 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:48.593255 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet d2ad471021a947e9a45d57c7be704453 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
W20260806 04:00:48.593974 32302 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 44752.1us
I20260806 04:00:48.605947 32741 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:48.613121 32322 catalog_manager.cc:5495] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:48.628541 2254 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:48.606019 2232 ts_tablet_manager.cc:1918] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:48.639456 2232 log.cc:1196] T bd55a7d68ee84cb7a69c962f984f9372 P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/bd55a7d68ee84cb7a69c962f984f9372
I20260806 04:00:48.652311 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet bd55a7d68ee84cb7a69c962f984f9372 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
W20260806 04:00:48.657496 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 69330.5us
I20260806 04:00:48.666442 1186 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:48.668449 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 81019.1us
W20260806 04:00:48.674547 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 47050.8us
I20260806 04:00:48.692514 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet a12b9e678e6049be86bca4e981f90379 with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:48.722107 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d7172fff185b4788acb609d2314e4b4b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260806 04:00:48.748777 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 72918.5us
W20260806 04:00:48.750036 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 70864.5us
I20260806 04:00:48.754597 32739 raft_consensus.cc:2932] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:48.793125 2254 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:48.797964 32320 catalog_manager.cc:5495] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:48.829675 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet d7172fff185b4788acb609d2314e4b4b with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:48.830677 2231 tablet_replica.cc:330] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:48.831748 2231 raft_consensus.cc:2229] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:48.832545 2231 raft_consensus.cc:2258] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:48.835681 2231 ts_tablet_manager.cc:1905] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:48.844794 2276 tablet_replica.cc:330] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:48.846040 2276 raft_consensus.cc:2229] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:48.846983 2276 raft_consensus.cc:2258] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:48.864099 2276 ts_tablet_manager.cc:1905] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:48.888254 2231 ts_tablet_manager.cc:1918] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 04:00:48.889757 2231 log.cc:1196] T d7172fff185b4788acb609d2314e4b4b P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/d7172fff185b4788acb609d2314e4b4b
I20260806 04:00:48.891661 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet d7172fff185b4788acb609d2314e4b4b (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:48.913369 2276 ts_tablet_manager.cc:1918] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:48.916337 2276 log.cc:1196] T a12b9e678e6049be86bca4e981f90379 P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/a12b9e678e6049be86bca4e981f90379
I20260806 04:00:48.919332 32609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:48.920110 32609 raft_consensus.cc:595] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:48.922276 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet a12b9e678e6049be86bca4e981f90379 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:48.923439 32609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:48.924018 32609 raft_consensus.cc:595] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:48.927960 32739 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:48.928537 32739 raft_consensus.cc:595] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:48.932281 32739 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db915af860484020942e9495eda28804"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:48.932830 32739 raft_consensus.cc:595] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:48.934988 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:00:48.957137 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:48.961707 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:49.041707 2252 raft_consensus.cc:979] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:49.042253 2252 raft_consensus.cc:1067] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:49.044065 32615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db915af860484020942e9495eda28804"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41324
I20260806 04:00:49.044617 32615 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:49.044943 32615 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.053222 32615 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:49.055940 32615 leader_election.cc:288] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:49.058588 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:49.059270 32467 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.066092 32467 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:49.070643 32553 leader_election.cc:302] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:49.073923 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:49.074532 32739 raft_consensus.cc:3032] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:49.074883 32739 raft_consensus.cc:729] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:49.075687 32739 consensus_queue.cc:260] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:49.076979 32739 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.080379 2027 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:49.080874 2027 raft_consensus.cc:686] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:49.081774 2027 consensus_queue.cc:237] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:49.086553 32739 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:49.104874 32321 catalog_manager.cc:5495] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:49.137581 2027 raft_consensus.cc:979] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:49.138221 2272 raft_consensus.cc:1067] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:49.139688 32469 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:00:49.140209 32469 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:49.140467 32469 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:49.145664 32469 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:49.148066 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:49.148664 32615 raft_consensus.cc:3032] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:00:49.148940 32615 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:49.149652 32615 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:49.150697 32615 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:49.148111 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:49.152704 32739 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:49.164368 32615 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:00:49.160853 32469 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:49.164368 32739 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:00:49.167716 32428 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:49.171609 2254 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:00:49.174582 2254 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:49.178210 2201 raft_consensus.cc:979] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:49.178717 2201 raft_consensus.cc:1067] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:49.175845 2254 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:49.180284 32470 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59514
I20260806 04:00:49.180928 32470 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:49.182420 32470 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.195724 32321 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 3 to 4, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 4 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:49.203759 32470 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:49.213665 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:49.214280 32746 raft_consensus.cc:3032] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:49.214596 32746 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:49.216698 32746 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:49.217962 32746 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.229350 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:49.230031 1186 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.235422 32470 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:49.235971 2269 raft_consensus.cc:979] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:49.237277 1186 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:49.238245 2272 raft_consensus.cc:1067] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:49.238514 32427 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:00:49.239841 32746 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:49.239925 2098 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:49.240505 2098 raft_consensus.cc:686] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:49.241384 2098 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:49.245605 1189 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37314
I20260806 04:00:49.246138 1189 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:49.246438 1189 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.255565 1189 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:49.258468 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:49.259037 32615 raft_consensus.cc:3032] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:49.259315 32615 raft_consensus.cc:729] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:49.259974 32615 consensus_queue.cc:260] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:49.260828 32615 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.264583 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:49.265159 32746 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:49.271556 32746 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:49.275782 1137 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:49.276677 2016 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:49.277168 2016 raft_consensus.cc:686] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:49.277199 32615 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:49.276753 1189 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:49.278718 32321 catalog_manager.cc:5495] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:49.278299 2016 consensus_queue.cc:237] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:49.343775 32321 catalog_manager.cc:5495] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:49.552119 32746 raft_consensus.cc:1261] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:49.553603 2193 consensus_queue.cc:1035] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:49.563686 32739 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:00:49.565330 2054 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:49.670261 32470 raft_consensus.cc:1261] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:49.677692 2269 consensus_queue.cc:1035] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:49.711041 32615 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:00:49.712574 2254 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:49.717761 32739 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:49.721511 2254 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:49.771392 1189 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:49.776186 32610 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:49.781138 2254 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:49.798024 2245 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:49.817521 32749 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:49.819258 2245 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:49.930773 32615 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:49.938514 32749 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:49.938839 32467 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:49.940040 2193 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:49.941118 2269 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 04:00:49.946310 32551 consensus_peers.cc:487] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f39a49c08914cb0a772f3187e29fe61. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:49.969647 2272 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:49.971253 32749 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.002061 32615 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.007112 32321 catalog_manager.cc:5495] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:50.009711 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:00:50.027170 32551 consensus_peers.cc:487] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9befcec6c17046459b03fa9ce98de94e. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:50.030155 32467 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.035753 32470 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:50.037367 2272 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:50.055815 2193 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.062038 32470 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.064266 32749 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:50.090348 32324 catalog_manager.cc:5495] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.093642 2272 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:00:50.111143 32470 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.126879 32739 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260806 04:00:50.128472 32427 consensus_peers.cc:487] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f44dd8bdcea4806ab5987c15e9bc5b5. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:50.129479 2289 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:50.142760 32615 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:50.149384 2254 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:50.155695 32749 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.171932 32470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:50.178864 32470 raft_consensus.cc:595] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:00:50.182350 2054 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.183527 32746 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.191321 32609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:50.191923 32609 raft_consensus.cc:595] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:50.201596 32322 catalog_manager.cc:5495] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.205063 32467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:50.205792 32467 raft_consensus.cc:595] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:00:50.214082 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260806 04:00:50.215775 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:50.216450 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:50.226534 2027 raft_consensus.cc:979] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:50.227029 2027 raft_consensus.cc:1067] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:50.228480 32610 consensus_queue.cc:237] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.231458 32467 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:00:50.232267 32467 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:50.234383 32467 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:50.266271 32615 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
W20260806 04:00:50.270023 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 50487.4us
I20260806 04:00:50.270833 2098 raft_consensus.cc:979] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:50.271569 2289 raft_consensus.cc:1067] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:50.274698 32746 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:50.276221 2027 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 04:00:50.280484 32551 consensus_peers.cc:487] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6023572bff444c85842d7b3351debf87. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:50.281054 32746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:00:50.282271 32746 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:50.282678 32746 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:50.283506 32469 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:50.284945 2027 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:50.310845 2027 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.315202 32467 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:50.317898 32748 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.320552 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:50.321229 32615 raft_consensus.cc:3032] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:00:50.321626 32615 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:50.322502 32615 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:50.323630 32615 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:50.323226 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:50.328017 32749 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:50.328372 32469 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.329731 32467 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:50.337083 32321 catalog_manager.cc:5495] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:50.338866 32746 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:50.341393 32749 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:00:50.344964 32746 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:50.344180 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:50.354856 32428 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:50.356509 2098 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:00:50.356982 2098 raft_consensus.cc:686] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:50.357908 32467 raft_consensus.cc:3032] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:50.358310 32467 raft_consensus.cc:729] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:50.358700 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:50.359320 32604 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:50.359138 32467 consensus_queue.cc:260] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:50.360390 32467 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:50.353744 32615 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:00:50.357828 2098 consensus_queue.cc:237] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:50.371546 32604 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:00:50.374554 32682 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:50.375834 2332 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:50.376379 2332 raft_consensus.cc:686] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:50.377365 2332 consensus_queue.cc:237] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:50.382750 2254 raft_consensus.cc:979] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:50.383285 2289 raft_consensus.cc:1067] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:50.385315 32746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:00:50.385907 32746 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:50.386304 32746 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:50.399226 32467 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:00:50.441094 32321 catalog_manager.cc:5495] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 2 to 3, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:50.447321 32746 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:50.456058 32746 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:50.457756 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:50.458340 1189 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:50.463897 2341 ts_tablet_manager.cc:927] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:50.467240 1189 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:00:50.470618 32682 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:50.470724 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:50.471729 2318 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:00:50.472277 2318 raft_consensus.cc:686] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:50.471341 32467 raft_consensus.cc:3032] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:00:50.473390 32467 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:50.474146 32467 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:50.473189 2318 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:50.475432 32467 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:50.486843 32467 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:00:50.504612 32321 catalog_manager.cc:5495] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:50.507684 2341 tablet_copy_client.cc:316] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:50.508473 2345 ts_tablet_manager.cc:927] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:50.510824 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 9befcec6c17046459b03fa9ce98de94e from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:50.511404 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 9befcec6c17046459b03fa9ce98de94e from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-9befcec6c17046459b03fa9ce98de94e
I20260806 04:00:50.521270 32631 tablet_copy_source_session.cc:205] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:50.524555 2345 tablet_copy_client.cc:316] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:50.537366 2341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9befcec6c17046459b03fa9ce98de94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:50.537246 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 2f39a49c08914cb0a772f3187e29fe61 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:50.537945 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 2f39a49c08914cb0a772f3187e29fe61 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-2f39a49c08914cb0a772f3187e29fe61
I20260806 04:00:50.545753 32631 tablet_copy_source_session.cc:205] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:50.549525 32321 catalog_manager.cc:5495] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 2 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:50.565027 2345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f39a49c08914cb0a772f3187e29fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:50.581420 2341 tablet_copy_client.cc:791] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:50.582073 2341 tablet_copy_client.cc:655] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:50.598011 2341 tablet_copy_client.cc:531] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:50.603794 2347 ts_tablet_manager.cc:927] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:50.612398 2341 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:50.617017 2347 tablet_copy_client.cc:316] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:50.619241 32505 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:50.619791 32505 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-4f44dd8bdcea4806ab5987c15e9bc5b5
I20260806 04:00:50.620534 2345 tablet_copy_client.cc:791] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:50.621117 2345 tablet_copy_client.cc:655] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:50.631738 32505 tablet_copy_source_session.cc:205] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:50.633625 2345 tablet_copy_client.cc:531] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:50.634866 2347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:50.653612 2345 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:50.666326 2341 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:50.667339 2341 tablet_bootstrap.cc:492] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:50.668102 2341 ts_tablet_manager.cc:1397] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.056s user 0.016s sys 0.005s
I20260806 04:00:50.678251 2341 raft_consensus.cc:348] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.681902 2341 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:50.683835 2347 tablet_copy_client.cc:791] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:50.684517 2347 tablet_copy_client.cc:655] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:50.688328 2347 tablet_copy_client.cc:531] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:50.687809 2341 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.705348 2341 ts_tablet_manager.cc:1428] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260806 04:00:50.709021 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-9befcec6c17046459b03fa9ce98de94e received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:50.709462 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-9befcec6c17046459b03fa9ce98de94e on tablet 9befcec6c17046459b03fa9ce98de94e with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:50.716928 2347 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:50.776675 2345 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:50.777684 2345 tablet_bootstrap.cc:492] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:50.778273 2345 ts_tablet_manager.cc:1397] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.125s user 0.020s sys 0.002s
I20260806 04:00:50.782626 2345 raft_consensus.cc:348] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.783459 2345 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:50.792135 2345 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.795609 2347 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:50.800475 32610 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:00:50.801446 2347 tablet_bootstrap.cc:492] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:50.802254 2347 ts_tablet_manager.cc:1397] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.086s user 0.019s sys 0.011s
I20260806 04:00:50.802295 2098 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:50.806080 2345 ts_tablet_manager.cc:1428] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260806 04:00:50.806995 2347 raft_consensus.cc:348] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.807952 2347 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:50.808626 2347 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:50.815346 2347 ts_tablet_manager.cc:1428] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260806 04:00:50.829846 32505 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-4f44dd8bdcea4806ab5987c15e9bc5b5 received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:50.830268 32505 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-4f44dd8bdcea4806ab5987c15e9bc5b5 on tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:50.834627 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-2f39a49c08914cb0a772f3187e29fe61 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:50.835071 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-2f39a49c08914cb0a772f3187e29fe61 on tablet 2f39a49c08914cb0a772f3187e29fe61 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:50.843142 2347 ts_tablet_manager.cc:927] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:50.856237 2347 tablet_copy_client.cc:316] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:50.861600 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 6023572bff444c85842d7b3351debf87 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:50.862157 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 6023572bff444c85842d7b3351debf87 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-6023572bff444c85842d7b3351debf87
I20260806 04:00:50.867692 1189 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:00:50.870534 32615 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:50.872473 32631 tablet_copy_source_session.cc:205] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:50.873234 2318 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:00:50.877377 2347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6023572bff444c85842d7b3351debf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:50.881310 2318 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:50.912065 32749 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:00:50.914099 2054 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260806 04:00:50.928658 32470 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:00:50.934589 1186 raft_consensus.cc:1203] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 04:00:50.942628 2318 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:00:51.008134 1189 raft_consensus.cc:1203] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 04:00:51.025605 32470 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:51.032213 2201 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:51.052448 1189 raft_consensus.cc:1203] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:51.058542 2347 tablet_copy_client.cc:791] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:51.089141 2347 tablet_copy_client.cc:655] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:51.117087 2347 tablet_copy_client.cc:531] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:51.161273 2347 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:51.206471 2347 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:51.207353 2347 tablet_bootstrap.cc:492] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:51.208050 2347 ts_tablet_manager.cc:1397] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.005s
I20260806 04:00:51.210769 2347 raft_consensus.cc:348] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:51.211805 2347 raft_consensus.cc:729] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:51.213629 2347 consensus_queue.cc:260] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:51.216413 2347 ts_tablet_manager.cc:1428] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260806 04:00:51.223027 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-6023572bff444c85842d7b3351debf87 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:51.223464 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-6023572bff444c85842d7b3351debf87 on tablet 6023572bff444c85842d7b3351debf87 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:51.432677 1189 raft_consensus.cc:1203] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:51.457424 2054 raft_consensus.cc:1050] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:51.460171 2054 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:51.476565 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:51.479157 32746 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:51.479676 1189 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:51.480944 2098 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:51.482278 2254 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:51.484004 32615 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:51.490466 2054 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:51.522359 2054 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:51.523972 32746 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:51.559062 2193 raft_consensus.cc:1050] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:51.565234 2193 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:51.607970 1189 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:51.546790 32610 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:51.626302 32322 catalog_manager.cc:5495] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:51.660917 2272 raft_consensus.cc:1050] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:51.663623 2272 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:51.673992 32746 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:51.674466 1186 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:51.676785 32467 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:51.691915 2027 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:51.694109 2027 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 04:00:51.745726 2392 log.cc:924] Time spent T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260806 04:00:51.745229 2374 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260806 04:00:51.781940 1186 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:51.791522 2193 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:51.803580 32746 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:51.807559 2272 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:51.817675 32469 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:51.824319 2393 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:51.866650 32470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:51.867300 32470 raft_consensus.cc:595] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:51.877207 32748 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:51.877882 32748 raft_consensus.cc:595] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:51.885296 32478 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:51.886694 32478 raft_consensus.cc:595] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:00:51.890309 32748 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:51.890946 32748 raft_consensus.cc:595] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:51.892338 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:00:51.894048 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:51.894673 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:00:51.929632 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 85349us
I20260806 04:00:51.956907 2193 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:51.963074 32746 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:51.982851 2254 raft_consensus.cc:979] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:51.983373 2254 raft_consensus.cc:1067] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:51.984047 2027 raft_consensus.cc:1050] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:51.989506 32615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41308
I20260806 04:00:51.990116 32615 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:51.990465 32615 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:00:51.990787 32467 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.016855 32615 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.020124 32615 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:52.029398 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:52.030020 32478 raft_consensus.cc:3032] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:00:52.030303 32478 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:52.031098 32478 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.030280 2411 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.032158 32478 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:00:52.033123 2027 consensus_queue.cc:237] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:52.038825 32322 catalog_manager.cc:5495] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.041615 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:52.042244 32748 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:00:52.049019 32748 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:00:52.054947 1186 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.061961 32478 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:00:52.069506 32553 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:52.070511 2409 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:00:52.071182 2409 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:52.072468 2409 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.089035 32746 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.056844 32469 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.166241 32467 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:00:52.170753 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 59636.3us
I20260806 04:00:52.167095 32321 catalog_manager.cc:5495] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.223385 32746 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:52.224506 2193 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:52.247843 32469 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:52.249468 2193 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:52.252709 32748 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:52.254184 2098 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:52.263625 32324 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 4 to 5, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 5 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:52.289857 2054 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.293689 32741 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.316744 2415 raft_consensus.cc:979] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:52.317301 2415 raft_consensus.cc:1067] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:52.318715 2386 raft_consensus.cc:979] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:52.318996 32478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59514
I20260806 04:00:52.319211 2386 raft_consensus.cc:1067] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:52.319587 32478 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:52.319934 32478 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.321432 32748 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:00:52.321952 32748 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:52.322249 32748 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.328661 2201 raft_consensus.cc:979] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:52.329123 2201 raft_consensus.cc:1067] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:52.329043 32478 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.330603 32467 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59514
I20260806 04:00:52.331117 32467 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:52.331470 32467 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.339396 32467 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.343690 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:52.344319 32749 raft_consensus.cc:3032] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:52.344605 32749 raft_consensus.cc:729] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:52.345274 32749 consensus_queue.cc:260] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.346122 32749 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.353987 32749 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:52.356684 32467 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:52.359333 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:52.359912 32739 raft_consensus.cc:3032] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:52.360237 32739 raft_consensus.cc:729] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:52.360949 32739 consensus_queue.cc:260] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.362004 32739 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.363686 32428 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:52.364658 32478 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:52.367015 2098 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:52.367568 2098 raft_consensus.cc:686] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:52.368690 2098 consensus_queue.cc:237] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.371708 1189 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.382131 1185 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:52.375190 1186 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:52.379699 2027 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.379935 32746 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.388307 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:52.394361 2435 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:52.401235 1187 raft_consensus.cc:2932] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.414485 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:52.411229 32324 catalog_manager.cc:5495] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:52.415134 32615 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.421468 32321 catalog_manager.cc:5495] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.438571 32615 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:52.439909 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:52.440577 32609 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.440362 32478 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.448576 32609 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:52.457042 32748 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:52.459750 32739 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:52.466827 32428 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:52.468044 2386 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:52.468827 2386 raft_consensus.cc:686] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:52.474006 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:52.481046 32469 raft_consensus.cc:3032] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:52.481554 32469 raft_consensus.cc:729] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:52.482424 32469 consensus_queue.cc:260] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:52.483587 32469 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.488821 2386 consensus_queue.cc:237] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:52.489737 2393 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:52.494895 32748 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:52.496448 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:52.497124 1189 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:52.512446 32604 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:52.478982 32324 catalog_manager.cc:5495] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:52.515758 32469 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:00:52.521690 32683 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:52.522629 2415 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:52.523631 2415 raft_consensus.cc:686] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:52.524893 2415 consensus_queue.cc:237] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:52.532977 32739 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:52.535704 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:52.537503 32739 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:00:52.532858 1189 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:00:52.538497 32321 catalog_manager.cc:5495] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:52.539953 2393 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:52.539047 32604 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:52.555754 1187 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:52.571590 2440 tablet_replica.cc:330] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:52.572552 2027 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:52.573064 2440 raft_consensus.cc:2229] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:52.574211 2440 raft_consensus.cc:2258] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:52.580932 32469 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:52.595798 2403 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:52.600095 2411 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.606566 1187 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:52.588510 2027 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260806 04:00:52.621599 1187 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.637653 32748 raft_consensus.cc:2932] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.654980 2193 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.660094 32478 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.663383 2403 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260806 04:00:52.682781 32322 catalog_manager.cc:5495] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:52.691727 2440 ts_tablet_manager.cc:1905] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:52.696694 1190 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.735956 1189 raft_consensus.cc:2932] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:52.750365 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 86537.9us
I20260806 04:00:52.749555 32321 catalog_manager.cc:5495] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 4 to 5, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.759220 2440 ts_tablet_manager.cc:1918] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 04:00:52.759817 2440 log.cc:1196] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/4f44dd8bdcea4806ab5987c15e9bc5b5
I20260806 04:00:52.764390 32478 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:00:52.771297 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9befcec6c17046459b03fa9ce98de94e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:52.731465 32322 catalog_manager.cc:5495] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:52.781050 2444 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:52.784920 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f39a49c08914cb0a772f3187e29fe61 with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:52.801563 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2f39a49c08914cb0a772f3187e29fe61 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:52.802369 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 4f44dd8bdcea4806ab5987c15e9bc5b5 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:52.814987 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 9befcec6c17046459b03fa9ce98de94e with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:52.822034 2456 tablet_replica.cc:330] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:52.833235 2456 raft_consensus.cc:2229] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:52.834350 2456 raft_consensus.cc:2258] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:52.856768 2454 tablet_replica.cc:330] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:52.857527 2456 ts_tablet_manager.cc:1905] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:52.857935 2454 raft_consensus.cc:2229] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:52.858745 2454 raft_consensus.cc:2258] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260806 04:00:52.891619 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 48647.3us
I20260806 04:00:52.905181 2454 ts_tablet_manager.cc:1905] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:52.913033 32746 raft_consensus.cc:1261] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:52.914520 2438 consensus_queue.cc:1035] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:52.929378 2456 ts_tablet_manager.cc:1918] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:52.933678 2456 log.cc:1196] T 9befcec6c17046459b03fa9ce98de94e P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/9befcec6c17046459b03fa9ce98de94e
I20260806 04:00:52.939052 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 9befcec6c17046459b03fa9ce98de94e (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:52.992126 2454 ts_tablet_manager.cc:1918] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 04:00:52.992663 2454 log.cc:1196] T 2f39a49c08914cb0a772f3187e29fe61 P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/2f39a49c08914cb0a772f3187e29fe61
I20260806 04:00:52.994334 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet 2f39a49c08914cb0a772f3187e29fe61 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:52.997370 32610 raft_consensus.cc:1261] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:53.003451 32610 consensus_queue.cc:237] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:53.006844 2054 consensus_queue.cc:1035] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:53.018711 32478 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:53.020172 2411 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260806 04:00:53.024699 2460 log.cc:924] Time spent T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.059s user 0.003s sys 0.003s
I20260806 04:00:53.026788 1189 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:53.028393 32604 raft_consensus.cc:1261] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:53.031706 32746 raft_consensus.cc:1261] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:53.033157 2438 consensus_queue.cc:1035] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:53.035236 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:53.037292 2409 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:53.039232 32478 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:53.107851 2098 consensus_queue.cc:1035] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:53.119791 1189 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:53.122007 2463 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 04:00:53.127497 2433 log.cc:924] Time spent T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.069s user 0.004s sys 0.004s
I20260806 04:00:53.148002 32473 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:53.165728 2463 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:53.201938 2411 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:53.209148 32322 catalog_manager.cc:5495] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:53.212431 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6023572bff444c85842d7b3351debf87 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:53.239936 1190 raft_consensus.cc:2932] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:53.278261 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 6023572bff444c85842d7b3351debf87 with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:53.279771 2456 tablet_replica.cc:330] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:53.281394 2456 raft_consensus.cc:2229] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:53.282637 2456 raft_consensus.cc:2258] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:53.291833 2456 ts_tablet_manager.cc:1905] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:53.432693 2456 ts_tablet_manager.cc:1918] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:53.433228 2456 log.cc:1196] T 6023572bff444c85842d7b3351debf87 P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/6023572bff444c85842d7b3351debf87
I20260806 04:00:53.435017 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 6023572bff444c85842d7b3351debf87 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:53.441847 32473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:53.442535 32473 raft_consensus.cc:595] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:53.445231 32746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:53.445783 32746 raft_consensus.cc:595] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:53.453428 32473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:53.455163 32473 raft_consensus.cc:595] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:00:53.457639 32610 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:53.458221 32610 raft_consensus.cc:595] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:53.459764 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:00:53.461505 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:53.462152 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:53.480350 2373 raft_consensus.cc:979] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:53.480948 2445 raft_consensus.cc:1067] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:53.487092 32604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41324
I20260806 04:00:53.487654 32604 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:53.487979 32604 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.488826 2444 raft_consensus.cc:979] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:53.489334 2444 raft_consensus.cc:1067] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:53.491092 32473 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:00:53.491662 32473 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:53.492051 32473 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.494776 32604 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:53.498111 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:53.498698 32748 raft_consensus.cc:3032] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:53.498986 32748 raft_consensus.cc:729] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:53.499683 32748 consensus_queue.cc:260] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:53.500757 32748 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.502475 32473 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:53.498384 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:53.504540 32469 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.507093 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:53.506923 32604 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:53.507647 32610 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:53.507925 32610 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:53.508659 32610 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:53.509703 32610 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.510300 32469 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:53.510596 32748 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:53.511396 32553 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:53.512627 2409 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:53.513093 2409 raft_consensus.cc:686] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:53.513900 2409 consensus_queue.cc:237] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:53.515622 32473 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:53.517231 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:53.517758 1189 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.542733 2054 raft_consensus.cc:979] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:53.543229 2054 raft_consensus.cc:1067] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:53.546433 32746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:00:53.546993 32746 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:53.547314 32746 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:53.562328 32746 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:53.565793 32610 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:53.566591 32321 catalog_manager.cc:5495] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:53.573164 1189 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:53.576112 32427 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:00:53.578707 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:53.579401 32473 raft_consensus.cc:3032] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:00:53.579720 32473 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:53.580585 32473 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:53.581729 32473 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:53.578886 2098 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:53.583369 2098 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:53.585438 2098 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:53.594367 32746 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:53.595528 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:53.596151 32615 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:53.597692 32473 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:00:53.598649 32683 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:53.599498 2373 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:00:53.599967 2373 raft_consensus.cc:686] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:53.600764 2373 consensus_queue.cc:237] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:53.602859 32615 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:00:53.656668 32321 catalog_manager.cc:5495] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 3 to 4, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 4 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:00:53.689121 32321 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:53.791067 2484 raft_consensus.cc:979] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:53.791579 2484 raft_consensus.cc:1067] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:53.793115 1189 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37290
I20260806 04:00:53.794924 1189 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:53.795284 1189 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.806869 1189 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:53.810096 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:53.810727 32469 raft_consensus.cc:3032] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:53.811008 32469 raft_consensus.cc:729] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:53.811663 32469 consensus_queue.cc:260] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:53.812741 32469 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.825047 1189 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:53.823937 32469 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:53.826862 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:53.827572 32610 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:53.829177 1137 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:00:53.830026 2295 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:53.848870 2295 raft_consensus.cc:686] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:53.849841 2295 consensus_queue.cc:237] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:53.865125 32610 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:00:53.869408 32321 catalog_manager.cc:5495] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:54.004946 32610 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:00:54.006762 2445 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:54.025626 32469 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:00:54.029309 2463 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:54.053977 32473 raft_consensus.cc:1261] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:54.057672 2027 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:54.063089 32748 raft_consensus.cc:1261] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:54.068940 2444 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:54.080991 32610 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:54.083830 2438 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:54.149931 1189 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:54.155895 2486 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 04:00:54.228127 2498 log.cc:924] Time spent T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260806 04:00:54.405671 32473 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:54.415553 2488 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:54.471851 32610 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:54.474426 2512 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:54.483054 32748 consensus_queue.cc:237] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:54.559648 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:00:54.563189 32682 consensus_peers.cc:487] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e78ed11baa324b6bbbf7a965a2bad24e. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:54.563485 32604 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:54.566875 2445 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:54.593078 32473 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:54.612735 2519 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:00:54.612239 2521 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.616230 32604 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.631404 32478 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:54.641954 32610 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:54.642541 32610 raft_consensus.cc:595] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:54.645948 32746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:54.646509 32746 raft_consensus.cc:595] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:54.648345 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:00:54.650055 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:54.650719 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:54.653741 32324 catalog_manager.cc:5495] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:54.667655 2521 raft_consensus.cc:979] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:54.668232 2445 raft_consensus.cc:1067] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:54.669992 32610 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41324
I20260806 04:00:54.670516 32610 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:54.670837 32610 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:54.695554 32748 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:54.695820 32610 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:54.698791 2386 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:54.699776 32610 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:00:54.697559 32427 consensus_peers.cc:487] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5a23c953ef64542931fb8ede0fcbb97. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:54.700381 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:54.700824 1190 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:54.701790 32615 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:00:54.704357 2486 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:54.707298 1190 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:54.708617 32551 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:54.708935 32473 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.712272 2409 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:54.712798 2409 raft_consensus.cc:686] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:54.713568 2409 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:54.720728 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:54.721385 32739 raft_consensus.cc:3032] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:54.721678 32739 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:54.722412 32739 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:54.723748 32739 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:54.734668 32739 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:54.752069 2486 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.757454 32615 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.756773 32748 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.775915 32322 catalog_manager.cc:5495] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:54.786098 32610 consensus_queue.cc:237] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:54.793083 32321 catalog_manager.cc:5495] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:54.807394 32739 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:00:54.808772 32473 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:00:54.810309 32551 consensus_peers.cc:487] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4f865676f884755987b8bd820489186. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:54.810849 2409 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:54.812372 2444 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:54.851219 2409 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.852859 32473 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.868310 32615 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:54.881326 32321 catalog_manager.cc:5495] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:54.910269 32748 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260806 04:00:54.910259 32551 consensus_peers.cc:487] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b2118a2ac1a4e3b873d360b7ef9b47e. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:00:54.911386 32473 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:00:54.912070 2409 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:00:54.913929 2444 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:00:54.917119 32739 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.971297 2503 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index -1 to 6, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:54.976377 32469 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index -1 to 6, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:55.006124 32322 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 6, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 5 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:55.029601 2506 raft_consensus.cc:979] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:55.030131 2506 raft_consensus.cc:1067] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:55.036803 32473 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:00:55.037395 32473 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:55.037752 32473 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:55.031071 32748 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index -1 to 6, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:00:55.068322 2542 ts_tablet_manager.cc:927] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:55.087368 32473 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:55.113075 2542 tablet_copy_client.cc:316] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:00:55.115600 32766 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet e78ed11baa324b6bbbf7a965a2bad24e from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:00:55.116110 32473 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:55.116227 32766 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet e78ed11baa324b6bbbf7a965a2bad24e from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-e78ed11baa324b6bbbf7a965a2bad24e
I20260806 04:00:55.130244 32766 tablet_copy_source_session.cc:205] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:55.135169 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78ed11baa324b6bbbf7a965a2bad24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:55.144398 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:55.145037 32615 raft_consensus.cc:3032] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:55.145328 32615 raft_consensus.cc:729] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:55.126055 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:55.148658 32739 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:55.146008 32615 consensus_queue.cc:260] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:55.150561 32615 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:55.168771 2546 ts_tablet_manager.cc:927] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:00:55.171702 32615 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:55.172925 32427 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:55.173815 2486 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:55.174300 2486 raft_consensus.cc:686] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:55.175169 2486 consensus_queue.cc:237] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:55.172199 32739 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:55.192278 2546 tablet_copy_client.cc:316] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:00:55.194975 32505 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet e5a23c953ef64542931fb8ede0fcbb97 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:00:55.195546 32505 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet e5a23c953ef64542931fb8ede0fcbb97 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-e5a23c953ef64542931fb8ede0fcbb97
I20260806 04:00:55.201238 2542 tablet_copy_client.cc:791] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:55.201844 2542 tablet_copy_client.cc:655] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:55.206372 32505 tablet_copy_source_session.cc:205] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:55.225785 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a23c953ef64542931fb8ede0fcbb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:55.227742 1189 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:55.230388 2506 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:55.227875 2542 tablet_copy_client.cc:531] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:55.255349 32322 catalog_manager.cc:5495] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:55.263335 2542 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:55.315173 2549 ts_tablet_manager.cc:927] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:55.310267 2546 tablet_copy_client.cc:791] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:55.317579 32739 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:55.327853 2027 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:55.330044 2546 tablet_copy_client.cc:655] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:55.338654 2549 tablet_copy_client.cc:316] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:55.341010 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet c4f865676f884755987b8bd820489186 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:55.341543 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet c4f865676f884755987b8bd820489186 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-c4f865676f884755987b8bd820489186
I20260806 04:00:55.346738 2546 tablet_copy_client.cc:531] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:55.349817 32631 tablet_copy_source_session.cc:205] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:55.355435 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f865676f884755987b8bd820489186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:55.377794 2546 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:55.409771 2542 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:55.410735 2542 tablet_bootstrap.cc:492] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:55.411418 2542 ts_tablet_manager.cc:1397] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.148s user 0.017s sys 0.011s
I20260806 04:00:55.419250 2549 tablet_copy_client.cc:791] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:55.422761 2549 tablet_copy_client.cc:655] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:55.424239 2542 raft_consensus.cc:348] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.436772 2549 tablet_copy_client.cc:531] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:55.436772 2566 ts_tablet_manager.cc:927] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:00:55.442679 2542 raft_consensus.cc:729] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:55.445336 2542 consensus_queue.cc:260] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.449875 2542 ts_tablet_manager.cc:1428] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.038s user 0.008s sys 0.001s
I20260806 04:00:55.451985 32766 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-e78ed11baa324b6bbbf7a965a2bad24e received from {username='slave'} at 127.0.0.1:33182
I20260806 04:00:55.456106 32766 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-e78ed11baa324b6bbbf7a965a2bad24e on tablet e78ed11baa324b6bbbf7a965a2bad24e with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:55.461032 2566 tablet_copy_client.cc:316] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:00:55.465377 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:00:55.465807 2549 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:55.465958 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 4b2118a2ac1a4e3b873d360b7ef9b47e from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-4b2118a2ac1a4e3b873d360b7ef9b47e
I20260806 04:00:55.486783 32631 tablet_copy_source_session.cc:205] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:00:55.493161 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:00:55.529776 2546 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:55.530946 2546 tablet_bootstrap.cc:492] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:55.531759 2546 ts_tablet_manager.cc:1397] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.154s user 0.032s sys 0.016s
I20260806 04:00:55.534580 2546 raft_consensus.cc:348] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.535480 2546 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:55.536648 2546 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.547724 2546 ts_tablet_manager.cc:1428] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260806 04:00:55.548120 2549 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:55.549919 2549 tablet_bootstrap.cc:492] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:55.551406 2549 ts_tablet_manager.cc:1397] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.086s user 0.027s sys 0.003s
I20260806 04:00:55.555742 2549 raft_consensus.cc:348] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.556737 2549 raft_consensus.cc:729] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:55.557435 2549 consensus_queue.cc:260] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.560405 2549 ts_tablet_manager.cc:1428] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260806 04:00:55.562151 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-c4f865676f884755987b8bd820489186 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:55.562536 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-c4f865676f884755987b8bd820489186 on tablet c4f865676f884755987b8bd820489186 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:55.563473 2566 tablet_copy_client.cc:791] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:00:55.564110 2566 tablet_copy_client.cc:655] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:00:55.577567 32505 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-e5a23c953ef64542931fb8ede0fcbb97 received from {username='slave'} at 127.0.0.1:55048
I20260806 04:00:55.578050 32505 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-e5a23c953ef64542931fb8ede0fcbb97 on tablet e5a23c953ef64542931fb8ede0fcbb97 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:00:55.592018 2566 tablet_copy_client.cc:531] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:00:55.602834 2566 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:00:55.604485 1190 raft_consensus.cc:1203] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 04:00:55.606594 32610 raft_consensus.cc:1261] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:55.621814 2565 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:55.662387 32746 raft_consensus.cc:1261] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:00:55.663827 2565 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:00:55.702031 1190 raft_consensus.cc:1203] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 04:00:55.715055 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:55.738754 2566 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:00:55.739665 2566 tablet_bootstrap.cc:492] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:00:55.769207 2566 ts_tablet_manager.cc:1397] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.167s user 0.037s sys 0.014s
I20260806 04:00:55.772003 2566 raft_consensus.cc:348] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.772908 2566 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:00:55.773555 2566 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.782595 2566 ts_tablet_manager.cc:1428] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260806 04:00:55.784192 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-4b2118a2ac1a4e3b873d360b7ef9b47e received from {username='slave'} at 127.0.0.1:42618
I20260806 04:00:55.784585 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-4b2118a2ac1a4e3b873d360b7ef9b47e on tablet 4b2118a2ac1a4e3b873d360b7ef9b47e with peer d4a238f055a44f459c62c224daa4d541
W20260806 04:00:55.797564 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 78792.4us
I20260806 04:00:55.855067 1190 raft_consensus.cc:1203] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:00:55.872407 32610 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:55.873040 32610 raft_consensus.cc:595] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:55.891180 32473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:55.891830 32473 raft_consensus.cc:595] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:00:55.904949 2027 raft_consensus.cc:979] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:55.906165 2503 raft_consensus.cc:1067] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:55.908208 32473 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:00:55.908856 32473 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:55.909240 32473 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:00:55.931459 1187 raft_consensus.cc:1203] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.5->[5.6-5.6] Dedup: 5.6->[]
I20260806 04:00:55.937726 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:00:55.939381 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:55.940001 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:55.954558 32473 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:55.956686 32473 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:55.967020 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:55.967669 32739 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:00:56.021034 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:56.021684 32610 raft_consensus.cc:3032] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:00:56.021981 32610 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:56.022658 32610 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:56.024276 32610 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:00:56.032732 32610 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:00:56.034277 32427 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:56.035218 2486 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:00:56.035697 2486 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:56.036580 2486 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:56.053916 32739 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:00:56.092866 32322 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 5 to 6, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 6 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:56.160755 2484 raft_consensus.cc:1050] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:56.163256 2484 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:56.172679 2517 raft_consensus.cc:1050] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:56.175282 2517 consensus_queue.cc:237] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:56.192636 1187 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:56.194250 2578 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:56.212947 32610 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:56.214754 2565 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:56.225764 2565 raft_consensus.cc:979] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:56.226306 2565 raft_consensus.cc:1067] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:56.227728 32739 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:00:56.228277 32739 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:56.228614 32739 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:56.236575 32739 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:56.239663 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:56.256760 32748 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:56.269268 32478 raft_consensus.cc:3032] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:56.269714 32478 raft_consensus.cc:729] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:56.270409 32478 consensus_queue.cc:260] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:56.271529 32478 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:56.311961 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:56.312657 1190 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:56.337545 1190 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:00:56.338871 32682 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:56.339753 2519 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:56.340256 2519 raft_consensus.cc:686] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:56.341096 2519 consensus_queue.cc:237] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:56.346633 2565 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:56.368132 32739 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:56.296736 2486 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:56.392185 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 262750us
I20260806 04:00:56.395051 32478 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
W20260806 04:00:56.516723 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 78509.8us
I20260806 04:00:56.302625 1187 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:56.669070 32748 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:56.692561 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 164368us
W20260806 04:00:56.795785 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 126259us
W20260806 04:00:56.812947 2391 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260806 04:00:56.828943 32478 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260806 04:00:56.840459 2592 log.cc:924] Time spent T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
W20260806 04:00:56.849243 2599 log.cc:924] Time spent T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.448s user 0.017s sys 0.015s
I20260806 04:00:56.852667 2637 raft_consensus.cc:1050] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:56.865921 2607 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:00:56.870424 32469 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:56.866887 2637 consensus_queue.cc:237] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:56.885176 1187 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:56.900048 32610 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:00:56.901255 32609 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:56.923153 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 110832us
W20260806 04:00:57.003785 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 96629.7us
I20260806 04:00:57.018389 2617 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:57.025377 32614 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:00:56.896413 2633 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.140s
I20260806 04:00:57.035575 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:57.051673 32739 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:00:57.021013 2638 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
W20260806 04:00:57.101295 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 108076us
W20260806 04:00:57.140430 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 89531.9us
W20260806 04:00:57.152547 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 150832us
I20260806 04:00:57.162566 2670 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:00:57.172041 32322 catalog_manager.cc:5495] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:57.117769 2603 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.083s
I20260806 04:00:57.209064 32321 catalog_manager.cc:5495] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260806 04:00:57.219967 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 69671us
I20260806 04:00:57.233709 2027 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:00:57.234267 2027 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:00:57.239367 2027 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:57.242087 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:00:57.249142 32478 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:57.249792 32478 raft_consensus.cc:595] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:00:57.276983 1186 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 5 LEARNER]: Advancing to term 6
I20260806 04:00:57.282647 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:00:57.286588 32553 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:00:57.291419 32478 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:57.292053 32478 raft_consensus.cc:595] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:00:57.301945 1187 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:57.309058 32470 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:00:57.309741 32470 raft_consensus.cc:595] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:57.313714 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260806 04:00:57.315129 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:57.315610 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:57.318137 1187 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:57.323385 2637 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 04:00:57.326653 2517 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:00:57.331936 1186 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:00:57.342051 32470 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:57.342542 32749 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:00:57.344177 2646 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:57.346009 2646 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:00:57.349153 2661 raft_consensus.cc:979] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:57.349598 2661 raft_consensus.cc:1067] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:57.361954 2666 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:00:57.366336 2628 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260806 04:00:57.370846 32746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:00:57.371460 32746 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:57.371793 32746 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:57.456152 32746 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:57.466493 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:57.467133 1189 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:57.517305 32746 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:57.520484 1189 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
W20260806 04:00:57.522927 2690 log.cc:924] Time spent T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260806 04:00:57.524556 32682 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:57.527489 2617 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:57.527979 2617 raft_consensus.cc:686] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:57.560567 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:57.561340 32467 raft_consensus.cc:3032] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:57.561759 32467 raft_consensus.cc:729] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:57.562630 32467 consensus_queue.cc:260] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:57.563999 32467 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:00:57.688988 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 96997.1us
I20260806 04:00:57.712568 32467 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
W20260806 04:00:57.716018 2711 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 04:00:57.665266 2617 consensus_queue.cc:237] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:00:57.799491 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 195992us
I20260806 04:00:57.855902 32739 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:57.777324 2675 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:57.910295 2694 log.cc:924] Time spent T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.206s user 0.001s sys 0.003s
I20260806 04:00:57.941370 32609 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:57.938184 2670 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:57.961853 32321 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39782 (request call id 296) took 1534 ms. Trace:
I20260806 04:00:57.967801 1187 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:57.962311 32321 rpcz_store.cc:276] 0806 04:00:56.427307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:00:56.427484 (+ 177us) service_pool.cc:227] Handling call
0806 04:00:56.839480 (+411996us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b5400150254. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d25cf2a 00007fd1731aed6c 00007fd1732a8c3e 00007fd1715bafb4 00007fd1715baf42 00007fd1715bae71 00007fd1715badec 00007fd1715ba0a2 00007fd16b67e94c 00007fd16b67dd86
0806 04:00:57.961700 (+1122220us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:00:57.294684 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:00:57.295137 (+ 453us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:00:57.295161 (+ 24us) write_op.cc:435] Acquired schema lock
0806 04:00:57.295176 (+ 15us) tablet.cc:660] Decoding operations
0806 04:00:57.295472 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:00:57.295513 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0806 04:00:57.295542 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:00:57.308412 (+ 12870us) tablet.cc:699] Row locks acquired
0806 04:00:57.311010 (+ 2598us) write_op.cc:260] PREPARE: finished
0806 04:00:57.312766 (+ 1756us) write_op.cc:270] Start()
0806 04:00:57.318079 (+ 5313us) write_op.cc:276] Timestamp: P: 1785988857312720 usec, L: 0
0806 04:00:57.321912 (+ 3833us) op_driver.cc:331] REPLICATION: starting
0806 04:00:57.361372 (+ 39460us) log.cc:842] Serialized 539 byte log entry
0806 04:00:57.771950 (+410578us) op_driver.cc:444] REPLICATION: finished
0806 04:00:57.782321 (+ 10371us) write_op.cc:301] APPLY: starting
0806 04:00:57.783571 (+ 1250us) tablet.cc:1365] starting BulkCheckPresence
0806 04:00:57.796006 (+ 12435us) tablet.cc:1368] finished BulkCheckPresence
0806 04:00:57.800156 (+ 4150us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:00:57.816297 (+ 16141us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:00:57.829126 (+ 12829us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:00:57.830935 (+ 1809us) write_op.cc:312] APPLY: finished
0806 04:00:57.844955 (+ 14020us) log.cc:842] Serialized 23 byte log entry
0806 04:00:57.878454 (+ 33499us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:00:57.888219 (+ 9765us) write_op.cc:454] Released schema lock
0806 04:00:57.888268 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0806 04:00:57.892764 (+ 4496us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:00:57.892799 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":261527,"reported_tablets":1,"spinlock_wait_cycles":540275072,"child_traces":[["op",{"apply.queue_time_us":7332,"apply.run_cpu_time_us":2856,"apply.run_wall_time_us":111406,"num_ops":1,"prepare.queue_time_us":62463,"prepare.run_cpu_time_us":2662,"prepare.run_wall_time_us":73915,"replication_time_us":446673}]]}
I20260806 04:00:58.013420 1186 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.024230 32478 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.094828 2731 raft_consensus.cc:979] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:58.095404 2731 raft_consensus.cc:1067] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:00:58.118121 32748 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:00:58.118696 32748 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:58.119042 32748 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:58.128959 32748 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.137854 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:58.138531 1190 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:58.137583 32322 catalog_manager.cc:5495] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.139348 32478 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.144364 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:58.144976 32474 raft_consensus.cc:3032] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:00:58.145243 32474 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:58.145923 32474 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.146996 32474 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:58.139797 32322 catalog_manager.cc:5495] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:00:58.154902 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 504842us
I20260806 04:00:58.167480 1190 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:00:58.180869 32682 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:58.198899 1187 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:58.201030 32478 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:58.172567 32748 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:58.200489 2517 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:00:58.177173 2731 raft_consensus.cc:1050] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:00:58.207494 2517 raft_consensus.cc:686] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:58.207314 2730 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:58.225767 2486 raft_consensus.cc:979] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:58.226536 2486 raft_consensus.cc:1067] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:58.229416 2519 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:58.220067 2517 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.245847 32474 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:00:58.255364 2731 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.321727 32324 catalog_manager.cc:5495] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:58.325403 32604 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:00:58.326495 32615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41308
I20260806 04:00:58.327484 32615 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:58.327957 32615 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:58.344774 32746 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:00:58.368907 1185 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:00:58.386173 2603 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:00:58.397830 2486 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:00:58.403398 32604 consensus_queue.cc:237] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.428056 2603 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:00:58.433665 32739 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:58.441118 2646 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260806 04:00:58.529371 1190 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:00:58.536439 2731 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Committing config change with OpId 6.8: config changed from index 6 to 8, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.559480 32748 consensus_queue.cc:237] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.560312 2503 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260806 04:00:58.574797 1190 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 6 to 8, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.576835 32604 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 6 to 8, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.582075 32615 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.584230 32615 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:58.585274 32471 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.594476 32324 catalog_manager.cc:5495] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 2 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:58.596832 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:58.597677 32469 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:00:58.598008 32469 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:58.598814 32469 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.600023 32469 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:58.606535 1187 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:58.612457 32609 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:58.613211 32469 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:00:58.628872 2730 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:00:58.646003 2611 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.651073 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:58.651757 1185 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:58.651243 1190 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.666946 1186 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 04:00:58.672578 2750 log.cc:924] Time spent T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
I20260806 04:00:58.679304 2722 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 04:00:58.688926 1185 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:00:58.692754 32324 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 6 to 8, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.700065 32553 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:00:58.701252 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 59818.3us
I20260806 04:00:58.704392 2646 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:00:58.704880 2646 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:58.705847 2646 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.708876 32749 raft_consensus.cc:2932] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.732940 2647 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260806 04:00:58.741977 2730 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.745162 32610 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.764604 32748 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 6 to 8, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.771575 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e78ed11baa324b6bbbf7a965a2bad24e with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:58.777069 1187 raft_consensus.cc:2932] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.818662 32749 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:58.820364 2731 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260806 04:00:58.823019 32475 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:00:58.824249 32322 catalog_manager.cc:5495] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.829218 2519 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:00:58.831559 1190 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:00:58.829218 32470 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:58.834440 2731 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260806 04:00:58.854688 32324 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 2 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 3 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:58.859923 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c4f865676f884755987b8bd820489186 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:58.873064 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f865676f884755987b8bd820489186 with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:58.903523 1187 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260806 04:00:58.908318 2731 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:00:58.909623 32322 catalog_manager.cc:5495] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 4 to 5, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.919505 32610 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260806 04:00:58.925268 2484 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.927872 32739 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.935338 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e5a23c953ef64542931fb8ede0fcbb97 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:58.926433 2565 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260806 04:00:58.940764 2486 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Committing config change with OpId 6.9: config changed from index 8 to 9, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.935490 1187 raft_consensus.cc:2932] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.985371 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet e78ed11baa324b6bbbf7a965a2bad24e with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59492
I20260806 04:00:58.988611 2770 tablet_replica.cc:330] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:58.990581 2770 raft_consensus.cc:2229] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:58.991583 2770 raft_consensus.cc:2258] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:58.994570 1185 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Committing config change with OpId 6.9: config changed from index 8 to 9, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:58.998963 32604 raft_consensus.cc:2932] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Committing config change with OpId 6.9: config changed from index 8 to 9, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:59.030053 32322 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 8 to 9, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 6 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:59.043282 32320 catalog_manager.cc:5495] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 4 to 5, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:00:59.051842 2770 ts_tablet_manager.cc:1905] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:59.061152 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:00:59.069756 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2118a2ac1a4e3b873d360b7ef9b47e with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33964
I20260806 04:00:59.071488 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a23c953ef64542931fb8ede0fcbb97 with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41296
I20260806 04:00:59.083120 2773 tablet_replica.cc:330] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:00:59.093602 2773 raft_consensus.cc:2229] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:59.095155 2772 tablet_replica.cc:330] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:00:59.096051 2770 ts_tablet_manager.cc:1918] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:00:59.095952 2773 raft_consensus.cc:2258] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:59.096664 2770 log.cc:1196] T c4f865676f884755987b8bd820489186 P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/c4f865676f884755987b8bd820489186
I20260806 04:00:59.100814 2773 ts_tablet_manager.cc:1905] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:59.106033 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet c4f865676f884755987b8bd820489186 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:59.107034 2770 tablet_replica.cc:330] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:00:59.098155 2772 raft_consensus.cc:2229] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:59.112941 2772 raft_consensus.cc:2258] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:59.117885 2772 ts_tablet_manager.cc:1905] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:59.111896 2770 raft_consensus.cc:2229] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 04:00:59.121799 2770 raft_consensus.cc:2258] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 04:00:59.131737 2770 ts_tablet_manager.cc:1905] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:00:59.155886 1185 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:00:59.157433 2669 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:00:59.162606 2773 ts_tablet_manager.cc:1918] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 04:00:59.163098 2773 log.cc:1196] T e5a23c953ef64542931fb8ede0fcbb97 P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/e5a23c953ef64542931fb8ede0fcbb97
I20260806 04:00:59.164875 2770 ts_tablet_manager.cc:1918] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 04:00:59.165266 2770 log.cc:1196] T e78ed11baa324b6bbbf7a965a2bad24e P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/e78ed11baa324b6bbbf7a965a2bad24e
I20260806 04:00:59.165078 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet e5a23c953ef64542931fb8ede0fcbb97 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:59.166675 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet e78ed11baa324b6bbbf7a965a2bad24e (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:59.167407 2772 ts_tablet_manager.cc:1918] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.8
I20260806 04:00:59.167829 2772 log.cc:1196] T 4b2118a2ac1a4e3b873d360b7ef9b47e P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/4b2118a2ac1a4e3b873d360b7ef9b47e
I20260806 04:00:59.169588 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 4b2118a2ac1a4e3b873d360b7ef9b47e (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:00:59.236547 32470 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:00:59.242769 2611 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:00:59.330431 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:59.455351 32748 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:59.456009 32748 raft_consensus.cc:595] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:00:59.458935 32748 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:59.459455 32748 raft_consensus.cc:595] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:59.461797 32748 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:00:59.462293 32748 raft_consensus.cc:595] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:00:59.465520 32604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:00:59.466058 32604 raft_consensus.cc:595] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:00:59.467968 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 04:00:59.473383 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:00:59.474102 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:00:59.735594 2611 raft_consensus.cc:979] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:59.736378 2611 raft_consensus.cc:1067] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:59.737972 32471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:00:59.738557 32471 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:59.738895 32471 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:59.746515 2670 raft_consensus.cc:979] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:59.747143 2722 raft_consensus.cc:1067] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:00:59.749240 32471 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.753412 32610 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41324
I20260806 04:00:59.753997 32610 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:59.754343 32610 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:59.768978 32610 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.771648 32610 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:59.771821 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:59.772562 32748 raft_consensus.cc:3032] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:59.772840 32748 raft_consensus.cc:729] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:59.773700 32748 consensus_queue.cc:260] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.774861 32471 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:00:59.775379 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:59.775942 1187 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:59.776876 32748 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:59.778625 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:00:59.779354 1185 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:59.782516 1187 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:00:59.783730 32427 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:00:59.785269 2486 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:00:59.785687 2486 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:00:59.786429 2486 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.786978 1185 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:59.787771 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:59.788928 32610 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:00:59.789248 32610 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:59.793578 32610 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.794751 32610 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:00:59.801178 32748 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:00:59.801522 32324 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 3 to 4, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 4 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:59.815145 32610 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:00:59.812650 32553 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:59.841516 2762 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:59.842041 2762 raft_consensus.cc:686] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:00:59.842875 2762 consensus_queue.cc:237] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.884860 2722 raft_consensus.cc:979] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:59.885319 2722 raft_consensus.cc:1067] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:00:59.886833 1187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37300
I20260806 04:00:59.887409 1187 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:59.887742 1187 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:59.889560 32320 catalog_manager.cc:5495] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:00:59.900468 1187 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.904230 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:59.904809 32748 raft_consensus.cc:3032] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:00:59.905092 32748 raft_consensus.cc:729] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:59.905822 32748 consensus_queue.cc:260] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.906900 32748 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:59.909714 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:00:59.910351 32471 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:00:59.913239 32748 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:00:59.914376 1137 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:00:59.915403 2488 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:00:59.915781 1187 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:59.915884 2488 raft_consensus.cc:686] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:00:59.917186 2488 consensus_queue.cc:237] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:00:59.940474 2670 raft_consensus.cc:979] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:59.940964 2670 raft_consensus.cc:1067] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:00:59.942665 32471 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:00:59.943192 32475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59514
I20260806 04:00:59.943745 32475 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:00:59.944059 32475 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:59.954442 32475 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:59.956792 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:00:59.957370 32604 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:59.966437 32604 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:00:59.969578 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:00:59.970191 32748 raft_consensus.cc:3032] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:00:59.970468 32748 raft_consensus.cc:729] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:00:59.971156 32748 consensus_queue.cc:260] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:00:59.972316 32748 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:00:59.975406 32475 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:00:59.976867 32427 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:00:59.984395 2790 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:00:59.991556 32321 catalog_manager.cc:5495] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 3, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 3 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:00:59.993556 32748 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:00.015367 2790 raft_consensus.cc:686] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:00.016409 2790 consensus_queue.cc:237] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:00.038265 32320 catalog_manager.cc:5495] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:00.156294 32475 consensus_queue.cc:237] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:00.165772 32748 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:00.168066 32604 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:01:00.171868 32427 consensus_peers.cc:487] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0201a31e3c594fdfa1cec8a9970becef. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:00.172971 2486 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:01:00.175275 2790 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:00.192564 2790 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.211926 32604 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.244690 32610 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:00.254968 32320 catalog_manager.cc:5495] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:00.269006 32748 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.274760 1187 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:00.283322 32739 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:00.284708 2611 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:00.284757 2762 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 04:01:00.291348 32551 consensus_peers.cc:487] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2611a58e8424bd088918c8aa63add87. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:00.297418 32475 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:00.301347 2762 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:01:00.311407 32604 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:01:00.312896 2794 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:00.336011 1187 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:01:00.337949 2794 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260806 04:01:00.338734 2784 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.393637 32739 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.397148 32748 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:00.396364 32471 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.423233 32749 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:00.430891 2793 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:00.413283 2669 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260806 04:01:00.435585 32475 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:00.442384 2793 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260806 04:01:00.451941 32739 raft_consensus.cc:1261] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:01:00.461532 2486 consensus_queue.cc:1035] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:00.445376 32321 catalog_manager.cc:5495] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.555620 32610 raft_consensus.cc:1261] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260806 04:01:00.583500 2809 log.cc:924] Time spent T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.099s user 0.001s sys 0.002s
I20260806 04:01:00.589250 2486 consensus_queue.cc:1035] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:00.548810 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:00.654577 32615 consensus_queue.cc:237] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:00.703794 32475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:01:00.704468 32475 raft_consensus.cc:595] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
W20260806 04:01:00.713567 2829 log.cc:924] Time spent T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20260806 04:01:00.713555 32614 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:01:00.714366 32614 raft_consensus.cc:595] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:00.717485 2839 ts_tablet_manager.cc:927] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:00.719653 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:01:00.721438 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:00.722055 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:00.762730 32475 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:00.766844 2811 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:00.769169 32746 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:01:00.770491 32551 consensus_peers.cc:487] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8feef3e400bd4fdda571ce5ead436f21. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:00.773885 2762 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:00.799060 2839 tablet_copy_client.cc:316] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:01:00.801118 32505 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet 0201a31e3c594fdfa1cec8a9970becef from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:01:00.801687 32505 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet 0201a31e3c594fdfa1cec8a9970becef from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-0201a31e3c594fdfa1cec8a9970becef
I20260806 04:01:00.810463 32505 tablet_copy_source_session.cc:205] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:01:00.804821 2811 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.818243 2839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0201a31e3c594fdfa1cec8a9970becef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:01:00.824815 32475 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.826073 32749 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.829530 2611 raft_consensus.cc:979] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:01:00.830252 2762 raft_consensus.cc:1067] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:01:00.835244 32471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:01:00.835777 32471 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:00.838047 32471 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:00.847543 32471 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:00.849884 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:00.850440 32615 raft_consensus.cc:3032] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:00.850888 32615 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:00.850914 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:00.851550 1186 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:00.851647 32615 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:00.852821 32615 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:00.863758 32320 catalog_manager.cc:5495] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:00.869948 32471 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:00.869134 1186 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:00.871723 32427 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:00.872891 2486 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:00.873342 2486 raft_consensus.cc:686] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:00.874148 2486 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:00.878731 2839 tablet_copy_client.cc:791] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:01:00.879357 2839 tablet_copy_client.cc:655] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:01:00.897104 2839 tablet_copy_client.cc:531] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:01:00.903865 32475 consensus_queue.cc:237] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:00.905879 32324 catalog_manager.cc:5495] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 2 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:00.908746 2790 raft_consensus.cc:979] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:01:00.909233 2790 raft_consensus.cc:1067] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:01:00.910713 32749 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33994
I20260806 04:01:00.911190 32749 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:00.911527 32749 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:00.921033 32749 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:00.924424 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:00.925019 32471 raft_consensus.cc:3032] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:00.926206 32471 raft_consensus.cc:729] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:00.926963 32471 consensus_queue.cc:260] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:00.928041 32471 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:00.917531 32615 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:00.937156 32749 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:00.940532 32471 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:00.940932 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:00.944803 32604 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:00.945394 32683 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:00.946242 2722 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:00.946738 2722 raft_consensus.cc:686] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:00.947638 2722 consensus_queue.cc:237] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:00.972985 32604 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
W20260806 04:01:00.976873 32427 consensus_peers.cc:487] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e870f9de1d6a4db4bbf149c6a785135c. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:00.977790 32749 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:00.978812 32604 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:00.982141 2839 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:01:00.986070 2794 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:00.988201 2596 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:01:01.002920 2850 ts_tablet_manager.cc:927] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:01.019457 2596 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:01.024549 2850 tablet_copy_client.cc:316] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:01:01.026386 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet c2611a58e8424bd088918c8aa63add87 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:01:01.026921 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet c2611a58e8424bd088918c8aa63add87 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-c2611a58e8424bd088918c8aa63add87
I20260806 04:01:01.013837 32324 catalog_manager.cc:5495] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 2 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:01:01.022367 32746 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:01.042970 32631 tablet_copy_source_session.cc:205] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:01:01.052261 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2611a58e8424bd088918c8aa63add87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:01:01.078723 2850 tablet_copy_client.cc:791] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:01:01.078495 32324 catalog_manager.cc:5495] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:01.079450 2850 tablet_copy_client.cc:655] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:01:01.084488 2850 tablet_copy_client.cc:531] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:01:01.094300 32604 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:01.103267 2850 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:01:01.133163 2839 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:01:01.134188 2839 tablet_bootstrap.cc:492] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:01:01.134996 2839 ts_tablet_manager.cc:1397] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.153s user 0.021s sys 0.003s
I20260806 04:01:01.142652 2839 raft_consensus.cc:348] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:01.154104 2839 raft_consensus.cc:729] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:01:01.157625 2850 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:01:01.158705 2850 tablet_bootstrap.cc:492] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:01:01.159509 2850 ts_tablet_manager.cc:1397] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.056s user 0.024s sys 0.005s
I20260806 04:01:01.154938 2839 consensus_queue.cc:260] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:01.162256 2839 ts_tablet_manager.cc:1428] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260806 04:01:01.162531 2850 raft_consensus.cc:348] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:01.163472 2850 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:01:01.164176 2850 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:01.166720 2850 ts_tablet_manager.cc:1428] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20260806 04:01:01.167305 32505 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-0201a31e3c594fdfa1cec8a9970becef received from {username='slave'} at 127.0.0.1:55048
I20260806 04:01:01.167721 32505 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-0201a31e3c594fdfa1cec8a9970becef on tablet 0201a31e3c594fdfa1cec8a9970becef with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:01:01.168326 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-c2611a58e8424bd088918c8aa63add87 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:01:01.168753 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-c2611a58e8424bd088918c8aa63add87 on tablet c2611a58e8424bd088918c8aa63add87 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:01:01.202776 2850 ts_tablet_manager.cc:927] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:01.228801 2850 tablet_copy_client.cc:316] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:01:01.230755 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 8feef3e400bd4fdda571ce5ead436f21 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:01:01.231287 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 8feef3e400bd4fdda571ce5ead436f21 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-8feef3e400bd4fdda571ce5ead436f21
I20260806 04:01:01.241348 32631 tablet_copy_source_session.cc:205] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:01:01.246946 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feef3e400bd4fdda571ce5ead436f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:01:01.267956 2850 tablet_copy_client.cc:791] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:01:01.268659 2850 tablet_copy_client.cc:655] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:01:01.292258 2850 tablet_copy_client.cc:531] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:01:01.311189 2850 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:01:01.341506 32604 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:01.343300 2790 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:01.368124 32471 raft_consensus.cc:1261] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:01.369593 2722 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:01.373992 2839 ts_tablet_manager.cc:927] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:01.396322 2850 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:01:01.397310 2850 tablet_bootstrap.cc:492] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:01:01.398061 2850 ts_tablet_manager.cc:1397] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.087s user 0.014s sys 0.013s
I20260806 04:01:01.414842 1187 raft_consensus.cc:1203] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:01:01.441926 2839 tablet_copy_client.cc:316] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.129:35941
I20260806 04:01:01.456650 2850 raft_consensus.cc:348] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:01.503553 2850 raft_consensus.cc:729] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:01:01.504297 2850 consensus_queue.cc:260] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:01.506842 2850 ts_tablet_manager.cc:1428] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.108s user 0.008s sys 0.000s
W20260806 04:01:01.491837 2865 log.cc:924] Time spent T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20260806 04:01:01.522388 32615 raft_consensus.cc:1261] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:01.523672 2722 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 04:01:01.528824 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 63342.4us
I20260806 04:01:01.532866 1186 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:01.538614 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-8feef3e400bd4fdda571ce5ead436f21 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:01:01.539108 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-8feef3e400bd4fdda571ce5ead436f21 on tablet 8feef3e400bd4fdda571ce5ead436f21 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:01:01.556365 32505 tablet_copy_service.cc:140] P ae0b3e43e7a64f789e51018fd5518d23: Received BeginTabletCopySession request for tablet e870f9de1d6a4db4bbf149c6a785135c from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:55048)
I20260806 04:01:01.556942 32505 tablet_copy_service.cc:161] P ae0b3e43e7a64f789e51018fd5518d23: Beginning new tablet copy session on tablet e870f9de1d6a4db4bbf149c6a785135c from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:55048: session id = d4a238f055a44f459c62c224daa4d541-e870f9de1d6a4db4bbf149c6a785135c
I20260806 04:01:01.615305 2794 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260806 04:01:01.707151 1189 raft_consensus.cc:1203] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:01:01.758277 32505 tablet_copy_source_session.cc:205] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:01:01.778716 2839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e870f9de1d6a4db4bbf149c6a785135c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:01:01.885530 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:01:02.021706 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 173062us
W20260806 04:01:02.050616 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 57792.1us
I20260806 04:01:02.090374 2839 tablet_copy_client.cc:791] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:01:02.091197 2839 tablet_copy_client.cc:655] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:01:02.099520 2839 tablet_copy_client.cc:531] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:01:02.105180 1190 raft_consensus.cc:1203] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260806 04:01:02.105544 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 378496us
I20260806 04:01:02.193378 2925 raft_consensus.cc:1050] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:01:02.203410 2925 consensus_queue.cc:237] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:02.218860 32746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:01:02.219476 32746 raft_consensus.cc:595] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:02.240419 32471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:01:02.241017 32471 raft_consensus.cc:595] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:02.241926 2839 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:01:02.242479 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:01:02.244932 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:02.245647 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:01:02.254508 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 144876us
I20260806 04:01:02.278395 2922 raft_consensus.cc:1050] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:01:02.283038 2922 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:02.290838 1190 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.296643 2722 raft_consensus.cc:979] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:01:02.297133 2910 raft_consensus.cc:1067] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:01:02.308411 32739 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.311117 2929 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:02.314945 2929 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:02.318591 32475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59514
I20260806 04:01:02.319224 32475 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:02.319650 32475 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:02.330401 32475 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:02.339990 32475 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:02.340934 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:02.341540 1186 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:02.343993 32614 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.348238 2879 raft_consensus.cc:979] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:01:02.348659 2879 raft_consensus.cc:1067] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:01:02.348937 1186 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:02.350023 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:02.350626 32749 raft_consensus.cc:3032] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:02.350881 32749 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:02.351562 32749 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:02.352414 32427 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:02.352612 32749 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:02.353284 2925 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:02.353729 2925 raft_consensus.cc:686] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:02.354559 2925 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:02.356451 2929 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 04:01:02.356489 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 49103.7us
I20260806 04:01:02.374356 32749 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:02.393757 32471 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.393896 32749 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.395867 2939 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:02.398350 2922 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:02.413466 1186 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.422832 2937 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:02.424335 32604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41308
I20260806 04:01:02.424811 32604 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:02.425057 32604 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:02.446710 32324 catalog_manager.cc:5495] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 3 to 4, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 4 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:02.457311 32604 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:02.461499 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:02.462173 32471 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:02.462574 32471 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:02.464260 32471 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:02.469749 32471 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:02.482599 32604 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:02.485086 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:02.486943 32749 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:02.492535 1190 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.489091 32471 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:02.498445 32553 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:02.498659 32749 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:02.499516 2874 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:02.500066 2874 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:02.501230 2874 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:02.489506 2931 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.507226 32615 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.511751 2937 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.562158 32745 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.562610 32320 catalog_manager.cc:5495] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.582716 32475 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.592800 1186 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:01:02.684249 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 63034.7us
W20260806 04:01:02.699662 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 87018.5us
I20260806 04:01:02.704450 2939 raft_consensus.cc:1050] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:01:02.706915 2939 consensus_queue.cc:237] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:02.716342 2839 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:01:02.717427 2839 tablet_bootstrap.cc:492] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:01:02.718042 2839 ts_tablet_manager.cc:1397] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.476s user 0.042s sys 0.000s
I20260806 04:01:02.729218 2839 raft_consensus.cc:348] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:02.730149 2839 raft_consensus.cc:729] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:01:02.730796 2839 consensus_queue.cc:260] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:02.744761 2839 ts_tablet_manager.cc:1428] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.026s user 0.009s sys 0.000s
I20260806 04:01:02.743921 32321 catalog_manager.cc:5495] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.738123 32322 catalog_manager.cc:5495] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:02.751010 32505 tablet_copy_service.cc:342] P ae0b3e43e7a64f789e51018fd5518d23: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-e870f9de1d6a4db4bbf149c6a785135c received from {username='slave'} at 127.0.0.1:55048
I20260806 04:01:02.751405 32505 tablet_copy_service.cc:433] P ae0b3e43e7a64f789e51018fd5518d23: ending tablet copy session d4a238f055a44f459c62c224daa4d541-e870f9de1d6a4db4bbf149c6a785135c on tablet e870f9de1d6a4db4bbf149c6a785135c with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:01:02.754318 32745 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.782912 32471 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.784237 1190 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.785593 2937 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:02.788229 2871 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:02.793212 32749 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:02.801120 2874 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:02.809687 1186 raft_consensus.cc:1203] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 04:01:02.824873 32749 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:01:02.826610 2794 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:02.886695 32478 consensus_queue.cc:237] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:02.892189 1190 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:01:02.894133 2931 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:02.901839 1186 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:01:02.902022 32749 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:01:02.902881 2871 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.904368 32749 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.906174 2794 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:02.909113 2794 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:02.915751 2879 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.919502 32749 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.921461 32610 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:02.927692 1186 raft_consensus.cc:2932] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.933254 32745 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:01:02.937021 2939 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:02.942139 1187 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:01:02.943161 32478 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.956434 2939 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.958525 32320 catalog_manager.cc:5495] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 3 to 4, VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.961081 2874 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260806 04:01:02.963006 1187 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:02.986282 32324 catalog_manager.cc:5495] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:02.979372 1185 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:03.001574 32749 raft_consensus.cc:2932] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:03.013851 32587 tablet_service.cc:1514] Processing DeleteTablet for tablet 0201a31e3c594fdfa1cec8a9970becef with delete_type TABLET_DATA_TOMBSTONED (TS dd7344d85a154539b7f8c484f2a76772 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41296
I20260806 04:01:03.019244 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0201a31e3c594fdfa1cec8a9970becef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260806 04:01:03.034245 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 51656.3us
I20260806 04:01:03.060263 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c2611a58e8424bd088918c8aa63add87 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:01:03.068887 2979 tablet_replica.cc:330] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: stopping tablet replica
I20260806 04:01:03.070094 2979 raft_consensus.cc:2229] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:01:03.073254 2979 raft_consensus.cc:2258] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:01:03.077337 2979 ts_tablet_manager.cc:1905] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:01:03.097625 2979 ts_tablet_manager.cc:1918] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:01:03.098208 2979 log.cc:1196] T 0201a31e3c594fdfa1cec8a9970becef P dd7344d85a154539b7f8c484f2a76772: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-1-root/wals/0201a31e3c594fdfa1cec8a9970becef
I20260806 04:01:03.101683 32302 catalog_manager.cc:4841] TS dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857): tablet 0201a31e3c594fdfa1cec8a9970becef (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:01:03.150197 32749 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:01:03.151876 2871 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 04:01:03.163064 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 117809us
I20260806 04:01:03.167562 32478 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:01:03.171602 2871 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:03.518226 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:01:03.700773 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 73665.9us
I20260806 04:01:03.724887 2925 raft_consensus.cc:1050] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:01:03.727304 2925 consensus_queue.cc:237] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:03.744503 32471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:01:03.745148 32471 raft_consensus.cc:595] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:01:03.750916 32614 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:01:03.754702 1186 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:03.756654 2978 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:03.759027 32749 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:03.762312 2931 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:03.762601 32604 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:03.772476 2978 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
W20260806 04:01:03.788311 1190 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.012s sys 0.014s
W20260806 04:01:03.812577 1190 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.637s user 0.013s sys 0.014s
I20260806 04:01:03.834259 32614 raft_consensus.cc:595] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:03.856072 32604 consensus_queue.cc:237] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:03.865006 1188 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:01:03.867720 32475 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:01:03.869482 2871 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:03.870657 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:01:03.872432 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:03.873116 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:03.890981 32478 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:03.887269 2871 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:01:03.961633 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 96337.5us
W20260806 04:01:03.968339 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 115955us
W20260806 04:01:03.967061 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 112959us
I20260806 04:01:03.999634 32322 catalog_manager.cc:5495] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.038513 2784 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260806 04:01:04.094707 2931 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.133730 2992 raft_consensus.cc:979] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:01:04.134425 2992 raft_consensus.cc:1067] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
W20260806 04:01:04.156694 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 44387.2us
W20260806 04:01:04.160847 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 70634.4us
W20260806 04:01:04.173492 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 106732us
W20260806 04:01:04.180902 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 63106.2us
I20260806 04:01:04.183228 1186 raft_consensus.cc:2932] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.193733 1190 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.201831 1188 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37290
I20260806 04:01:04.202493 1188 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:04.202840 1188 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:04.210788 2925 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:04.211400 2925 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:04.213948 2925 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:04.241158 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 70269.9us
I20260806 04:01:04.244570 2922 raft_consensus.cc:979] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: : Instructing follower ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:01:04.246233 2922 raft_consensus.cc:1067] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Signalling peer ae0b3e43e7a64f789e51018fd5518d23 to start an election
I20260806 04:01:04.250926 1188 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:04.253937 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 67829.1us
I20260806 04:01:04.255471 32302 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8feef3e400bd4fdda571ce5ead436f21 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:01:04.257474 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:04.258134 32603 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.260352 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:04.267397 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:04.269891 32745 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.274097 1188 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:04.275785 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:04.276326 32615 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:04.281136 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:04.281890 32474 raft_consensus.cc:3032] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:04.282220 32474 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:04.283113 32474 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:04.285789 32474 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:04.290899 32615 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:04.294698 1136 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:04.295801 2955 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:04.296339 2955 raft_consensus.cc:686] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:04.297189 2955 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:04.307564 32427 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:04.313755 3017 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:01:04.328002 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 94602us
I20260806 04:01:04.331570 32471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:59498
I20260806 04:01:04.332160 32471 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:04.332505 32471 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:04.340885 32471 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:04.343814 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:04.344595 32746 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:04.342686 32471 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:04.350266 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:04.350920 32474 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:04.351778 32603 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:04.352308 32603 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:04.353098 32603 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:04.354202 32603 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:04.366895 32603 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:04.367944 32427 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:04.368649 2931 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:04.369246 2931 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:04.370234 2931 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:04.373796 32746 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:04.383657 32321 catalog_manager.cc:5495] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 reported cstate change: config changed from index 3 to 4, VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) evicted. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.388379 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet 8feef3e400bd4fdda571ce5ead436f21 with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:01:04.415010 32465 tablet_service.cc:1514] Processing DeleteTablet for tablet c2611a58e8424bd088918c8aa63add87 with delete_type TABLET_DATA_TOMBSTONED (TS ae0b3e43e7a64f789e51018fd5518d23 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59492
I20260806 04:01:04.420960 3038 tablet_replica.cc:330] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:01:04.422725 3038 raft_consensus.cc:2229] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:01:04.431968 3038 raft_consensus.cc:2258] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:01:04.441568 3038 ts_tablet_manager.cc:1905] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:01:04.453104 3039 tablet_replica.cc:330] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: stopping tablet replica
I20260806 04:01:04.429811 32322 catalog_manager.cc:5495] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:04.460040 3039 raft_consensus.cc:2229] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:01:04.461107 3039 raft_consensus.cc:2258] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:01:04.464511 3039 ts_tablet_manager.cc:1905] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:01:04.464883 32322 catalog_manager.cc:5495] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 2 to 3, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:04.497586 32321 catalog_manager.cc:5495] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 3 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:04.501437 3038 ts_tablet_manager.cc:1918] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:01:04.502245 3038 log.cc:1196] T 8feef3e400bd4fdda571ce5ead436f21 P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/8feef3e400bd4fdda571ce5ead436f21
I20260806 04:01:04.506784 32471 consensus_queue.cc:237] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:04.508476 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet 8feef3e400bd4fdda571ce5ead436f21 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:01:04.524672 1192 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:01:04.526319 2931 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:01:04.526747 32603 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 04:01:04.528731 3017 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:01:04.534863 3039 ts_tablet_manager.cc:1918] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:01:04.535501 3039 log.cc:1196] T c2611a58e8424bd088918c8aa63add87 P ae0b3e43e7a64f789e51018fd5518d23: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-0-root/wals/c2611a58e8424bd088918c8aa63add87
I20260806 04:01:04.544068 32303 catalog_manager.cc:4841] TS ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941): tablet c2611a58e8424bd088918c8aa63add87 (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:01:04.564546 3022 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.575408 32608 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.589139 1192 raft_consensus.cc:2932] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:01:04.608978 32526 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdae0 after lost signal to thread 30649
W20260806 04:01:04.610707 32526 debug-util.cc:398] Leaking SignalData structure 0x7b0800060f80 after lost signal to thread 32373
W20260806 04:01:04.612821 32526 debug-util.cc:398] Leaking SignalData structure 0x7b0800140d60 after lost signal to thread 32530
W20260806 04:01:04.616991 32526 debug-util.cc:398] Leaking SignalData structure 0x7b080026f480 after lost signal to thread 32659
W20260806 04:01:04.618580 32526 debug-util.cc:398] Leaking SignalData structure 0x7b0800252840 after lost signal to thread 326
W20260806 04:01:04.627640 32526 debug-util.cc:398] Leaking SignalData structure 0x7b080036a100 after lost signal to thread 1231
I20260806 04:01:04.663182 32326 catalog_manager.cc:5495] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 3 to 4, VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131) evicted. New cstate: current_term: 1 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:04.675105 32303 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e870f9de1d6a4db4bbf149c6a785135c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 04:01:04.679514 32723 tablet_service.cc:1514] Processing DeleteTablet for tablet e870f9de1d6a4db4bbf149c6a785135c with delete_type TABLET_DATA_TOMBSTONED (TS f7c202b562ce482db283463d7f02e136 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33964
I20260806 04:01:04.683710 3038 tablet_replica.cc:330] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: stopping tablet replica
I20260806 04:01:04.684924 3038 raft_consensus.cc:2229] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:01:04.685657 3038 raft_consensus.cc:2258] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:01:04.689831 3038 ts_tablet_manager.cc:1905] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 04:01:04.742445 3038 ts_tablet_manager.cc:1918] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 04:01:04.742825 3038 log.cc:1196] T e870f9de1d6a4db4bbf149c6a785135c P f7c202b562ce482db283463d7f02e136: Deleting WAL directory at /tmp/dist-test-task57efHF/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988747097609-30646-0/minicluster-data/ts-2-root/wals/e870f9de1d6a4db4bbf149c6a785135c
I20260806 04:01:04.744280 32303 catalog_manager.cc:4841] TS f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975): tablet e870f9de1d6a4db4bbf149c6a785135c (table test-workload [id=1530919f0c614d53a2009fb4cfbfa791]) successfully deleted
I20260806 04:01:04.817142 32474 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:04.818658 2955 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:04.844285 32608 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:04.857306 2954 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:04.886643 32746 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:04.888098 3022 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:04.909350 32608 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:04.911440 2794 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:05.040676 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:05.097414 32474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:01:05.098083 32474 raft_consensus.cc:595] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:05.100051 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260806 04:01:05.101786 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:05.102372 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:01:05.419437 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 91891.8us
W20260806 04:01:05.456804 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 56030.3us
I20260806 04:01:05.473381 2794 raft_consensus.cc:979] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: : Instructing follower dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:01:05.474246 2794 raft_consensus.cc:1067] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Signalling peer dd7344d85a154539b7f8c484f2a76772 to start an election
I20260806 04:01:05.476879 32603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:41308
I20260806 04:01:05.477401 32603 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:05.477748 32603 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:05.530956 32603 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:05.533353 32603 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:05.536180 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:05.536435 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:05.536799 32475 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:05.537068 32749 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:05.537161 32475 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:05.537966 32475 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:05.539142 32475 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:05.543763 32749 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:05.545073 32553 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:05.546268 3047 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:05.546734 3047 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:05.549921 3047 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:05.561017 32475 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:05.571269 32326 catalog_manager.cc:5495] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 3 to 4, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 4 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:05.783222 32608 consensus_queue.cc:237] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:05.822084 32749 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:01:05.823542 2874 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 04:01:05.826800 32551 consensus_peers.cc:487] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7c0dfae3c464cd18cfd95401b49eed9. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:05.823515 32475 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:01:05.838794 3031 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:05.845963 3031 raft_consensus.cc:2932] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:05.858381 32471 raft_consensus.cc:2932] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:05.866788 32745 consensus_queue.cc:237] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:05.869196 32749 raft_consensus.cc:2932] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:05.877359 32326 catalog_manager.cc:5495] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index -1 to 3, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:05.891860 32608 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 04:01:05.892000 32682 consensus_peers.cc:487] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4533f141e644c719df87540a1ca54e8. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:05.890739 32471 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:05.900796 2989 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:05.908003 3012 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:05.935127 32474 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:01:05.936802 2874 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:05.943743 32746 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:01:05.945179 2874 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:01:06.088842 3049 raft_consensus.cc:2932] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.093868 32608 raft_consensus.cc:2932] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.114332 32608 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:06.120970 32475 raft_consensus.cc:2932] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
W20260806 04:01:06.126369 32551 consensus_peers.cc:487] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d24bcc94fb734c95849f4cb89c15ca8e. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:06.126312 32745 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 04:01:06.127959 3031 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:06.123746 32321 catalog_manager.cc:5495] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:06.138509 3029 raft_consensus.cc:2932] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.148594 32745 raft_consensus.cc:2932] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.156217 32475 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 04:01:06.158041 2874 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:06.162295 32475 raft_consensus.cc:2932] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.173528 32608 consensus_queue.cc:237] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:01:06.181475 32551 consensus_peers.cc:487] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12e7971e809846ada28ebf1d6cea4837. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:06.183205 32326 catalog_manager.cc:5495] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 5, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 4 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.184824 32475 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:06.186672 32745 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:01:06.188897 3031 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:06.190055 2874 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:06.192638 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:06.218396 2874 raft_consensus.cc:2932] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.229710 32475 raft_consensus.cc:2932] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.226377 32745 raft_consensus.cc:2932] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.253072 32321 catalog_manager.cc:5495] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: config changed from index -1 to 2, NON_VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132) added. New cstate: current_term: 1 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:06.269556 3088 ts_tablet_manager.cc:927] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:06.273080 32603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4f865676f884755987b8bd820489186"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:01:06.273753 32603 raft_consensus.cc:595] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:01:06.276554 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260806 04:01:06.278551 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:06.279210 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:06.301666 3088 tablet_copy_client.cc:316] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:01:06.317593 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet d7c0dfae3c464cd18cfd95401b49eed9 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:01:06.318146 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet d7c0dfae3c464cd18cfd95401b49eed9 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-d7c0dfae3c464cd18cfd95401b49eed9
I20260806 04:01:06.326042 32631 tablet_copy_source_session.cc:205] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:01:06.336138 3088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c0dfae3c464cd18cfd95401b49eed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:01:06.360733 3088 tablet_copy_client.cc:791] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:01:06.361327 3088 tablet_copy_client.cc:655] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:01:06.367038 3088 tablet_copy_client.cc:531] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:01:06.375938 3088 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:01:06.433398 3088 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:01:06.434315 3088 tablet_bootstrap.cc:492] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:01:06.435002 3088 ts_tablet_manager.cc:1397] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 0.059s user 0.034s sys 0.022s
I20260806 04:01:06.438199 3088 raft_consensus.cc:348] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:06.439106 3088 raft_consensus.cc:729] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:01:06.439792 3088 consensus_queue.cc:260] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:06.442349 3088 ts_tablet_manager.cc:1428] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 04:01:06.454067 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-d7c0dfae3c464cd18cfd95401b49eed9 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:01:06.454479 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-d7c0dfae3c464cd18cfd95401b49eed9 on tablet d7c0dfae3c464cd18cfd95401b49eed9 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:01:06.725008 3098 ts_tablet_manager.cc:927] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:06.732331 3098 tablet_copy_client.cc:316] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.131:34975
I20260806 04:01:06.734171 32766 tablet_copy_service.cc:140] P f7c202b562ce482db283463d7f02e136: Received BeginTabletCopySession request for tablet e4533f141e644c719df87540a1ca54e8 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:33182)
I20260806 04:01:06.734781 32766 tablet_copy_service.cc:161] P f7c202b562ce482db283463d7f02e136: Beginning new tablet copy session on tablet e4533f141e644c719df87540a1ca54e8 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:33182: session id = d4a238f055a44f459c62c224daa4d541-e4533f141e644c719df87540a1ca54e8
I20260806 04:01:06.747332 32766 tablet_copy_source_session.cc:205] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:01:06.874396 3098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4533f141e644c719df87540a1ca54e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:01:07.455667 1192 raft_consensus.cc:1203] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 04:01:07.410096 3049 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:07.465850 2874 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:07.466848 2874 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:07.464046 3049 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:07.479728 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 43719.2us
I20260806 04:01:07.488811 32322 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36880 (request call id 228) took 1037 ms. Trace:
I20260806 04:01:07.489634 32322 rpcz_store.cc:276] 0806 04:01:06.451105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:06.451274 (+ 169us) service_pool.cc:227] Handling call
0806 04:01:07.488709 (+1037435us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260806 04:01:07.453251 2794 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:07.511329 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 46441.9us
I20260806 04:01:07.640413 3116 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:07.641418 3116 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:07.751585 3113 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:07.752872 3113 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:07.757635 3113 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:07.801482 3098 consensus_meta.cc:287] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.524s user 0.013s sys 0.001s
I20260806 04:01:07.825160 3123 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:07.826632 3123 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:07.914465 3116 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:07.915838 3116 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:07.917656 3116 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:07.803865 3098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37300 (request call id 524) took 1261 ms. Trace:
I20260806 04:01:07.947875 3116 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:07.945443 2874 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:07.933490 3123 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:07.948511 3098 rpcz_store.cc:276] 0806 04:01:06.542343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:06.542574 (+ 231us) service_pool.cc:227] Handling call
0806 04:01:06.725365 (+182791us) ts_tablet_manager.cc:928] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
0806 04:01:06.873201 (+147836us) tablet_copy_client.cc:361] Tablet copy session begun
0806 04:01:07.275833 (+402632us) tablet_metadata.cc:641] Metadata flushed
0806 04:01:07.276473 (+ 640us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 04:01:07.802415 (+525942us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 04:01:07.803003 (+ 588us) tablet_replica.cc:205] Creating consensus instance
0806 04:01:07.803595 (+ 592us) ts_tablet_manager.cc:983] Replica registered.
0806 04:01:07.803722 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":252,"tablet-copy.queue_time_us":181883,"thread_start_us":189101,"threads_started":2}
W20260806 04:01:07.952445 32551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.016s sys 0.017s
W20260806 04:01:07.953150 32551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.506s user 0.016s sys 0.017s
W20260806 04:01:07.954717 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.000s sys 0.032s
W20260806 04:01:07.955300 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.516s user 0.000s sys 0.032s
W20260806 04:01:07.955879 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 521876us
W20260806 04:01:07.956207 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 443079us
I20260806 04:01:07.951830 3098 tablet_copy_client.cc:791] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
W20260806 04:01:07.971067 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 471389us
I20260806 04:01:07.981299 3144 ts_tablet_manager.cc:927] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:07.991852 3098 tablet_copy_client.cc:655] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:01:07.998404 3012 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:07.998764 2874 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.010133 3123 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.005651 3142 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.022629 3123 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:08.023939 3142 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.026676 3123 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:08.028116 3140 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.029090 3140 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.031687 3134 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.032649 3134 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:08.016361 3012 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.006223 3135 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.068161 3135 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:08.069813 3140 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:08.073073 3134 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:08.074545 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 534997us
I20260806 04:01:08.031073 3142 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:08.076174 3012 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:08.099669 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 100258us
I20260806 04:01:08.108518 3135 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:08.019920 2874 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.131906 3140 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.132151 3158 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.136548 2874 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:08.135492 3158 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:08.137655 2874 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.138577 2874 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:08.143292 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:08.143734 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:08.147295 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:08.152004 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:08.190413 3165 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.183674 3166 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.219373 3167 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.637419 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:08.944859 1186 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:08.943902 3184 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.702353 3169 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.181327 2874 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:07.552709 3031 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:07.511217 2794 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.088617 3031 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.308164 3049 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.091650 3031 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:08.692761 3117 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:08.624768 3123 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.155599 3134 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.165776 3123 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.166821 3134 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.169050 3134 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:08.135134 3140 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.172287 3140 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:08.152069 3142 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.174151 3142 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.704739 3160 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:08.169699 3144 tablet_copy_client.cc:316] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:01:08.156729 3148 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.152518 3150 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.194938 3212 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:08.147876 3152 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.144351 3158 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:08.760864 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:08.764052 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:08.766674 3172 raft_consensus.cc:979] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
W20260806 04:01:08.769073 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.778s user 0.027s sys 0.044s
W20260806 04:01:08.821452 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.014s sys 0.005s
I20260806 04:01:08.825567 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:08.830672 1192 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.361s user 0.022s sys 0.000s
I20260806 04:01:08.831463 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:08.832388 32652 debug-util.cc:398] Leaking SignalData structure 0x7b080032e840 after lost signal to thread 30649
I20260806 04:01:08.832306 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:08.835050 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:08.841786 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:08.845857 32427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.006s sys 0.004s
I20260806 04:01:08.848459 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:08.845335 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:08.844030 3178 ts_tablet_manager.cc:927] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:08.861030 3174 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.867276 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:09.221699 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.098s user 0.015s sys 0.005s
I20260806 04:01:08.874405 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:08.877918 3114 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.881445 3138 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.891213 3170 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.892380 3113 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.894804 3163 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.899485 3183 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.905215 3176 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:08.906462 3175 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.906459 3173 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:08.914326 3104 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:08.944293 3165 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:08.673383 32558 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.004s sys 0.005s
I20260806 04:01:08.949536 3187 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.951393 3188 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:08.943993 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:08.944876 3167 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.945220 3166 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:08.971697 32683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.461s user 0.005s sys 0.015s
I20260806 04:01:08.965525 3186 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:08.976393 3191 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:08.973506 3169 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.972617 3184 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.050292 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:08.978797 3193 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.062206 3195 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.065301 3199 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.066602 3200 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.195611 3212 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.067850 3198 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.080322 3197 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.084830 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 1979) took 1660 ms. Trace:
I20260806 04:01:08.151969 3116 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.364730 3116 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.107963 3203 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.367198 3169 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.091419 2794 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.111752 3049 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.100378 3117 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:09.148516 3201 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:09.371119 3178 tablet_copy_client.cc:316] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: tablet copy: Beginning tablet copy session from remote peer at address 127.29.237.130:37857
I20260806 04:01:09.370923 3049 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:09.153769 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 56967.4us
I20260806 04:01:09.155154 3209 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:09.164817 3208 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:08.149504 3135 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.169312 3123 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.177224 3160 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.375617 3135 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:09.190725 3142 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.194272 3218 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.189700 3148 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.203722 3222 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.195381 3150 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.205183 3204 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.205046 3210 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.383116 3218 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:08.869504 3179 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:09.221781 1192 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.752s user 0.023s sys 0.000s
W20260806 04:01:09.223131 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.234s user 0.029s sys 0.045s
I20260806 04:01:09.204487 3206 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:09.213582 3152 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.228971 3172 raft_consensus.cc:1067] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
W20260806 04:01:09.225847 32427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.217s user 0.009s sys 0.004s
W20260806 04:01:09.237052 32558 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.080s user 0.005s sys 0.006s
I20260806 04:01:09.287580 1185 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
W20260806 04:01:09.290338 32683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.798s user 0.006s sys 0.017s
I20260806 04:01:09.229637 3163 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.292766 3224 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.294279 3167 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.234956 3173 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.235652 3140 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.235208 3187 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.295621 3200 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.310067 3230 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:09.310750 3212 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.314059 3232 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.235459 3113 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.264622 3217 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.289523 3174 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.308145 3219 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.299599 32739 rpcz_store.cc:276] 0806 04:01:07.424017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:07.429307 (+ 5290us) service_pool.cc:227] Handling call
0806 04:01:07.429720 (+ 413us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:07.508796 (+ 79076us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:07.508843 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:07.508901 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:07.508946 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:09.084197 (+1575251us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b54000f9e54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:09.084259 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:09.084558 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":181145856}
I20260806 04:01:09.323148 3104 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:09.337918 32652 debug-util.cc:398] Leaking SignalData structure 0x7b080030f000 after lost signal to thread 32373
I20260806 04:01:09.310876 3223 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.303249 3114 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.235265 3175 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.316109 3198 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.310730 3186 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.306865 3170 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.306556 3188 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.344789 3225 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.319888 3138 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.329389 3183 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.328677 3176 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:09.337698 3195 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.362578 3184 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.350081 3166 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.330540 3197 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.341316 3220 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.337158 3193 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.316118 3199 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.367508 3203 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.338059 3191 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.371387 3201 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.375425 3208 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.184141 3215 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.370527 3165 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.379460 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.379258 3209 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.375897 3117 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.381384 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.354645 3231 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.383692 3160 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:09.387470 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.39816e+06us
I20260806 04:01:09.385766 3204 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.431958 32682 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:09.433765 3233 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.435248 3116 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.437507 3135 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.380762 3049 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.445957 3187 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:09.447889 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 89488.5us
I20260806 04:01:09.446010 3173 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.435846 3148 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.436357 3218 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.487489 3163 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.386688 1184 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:09.451920 3217 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.385018 3234 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.461242 3150 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.386051 3222 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.458050 3232 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.487581 32478 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:09.474467 3195 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.490679 2794 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.491410 3054 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.471416 3184 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:09.447002 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 1.95084e+06us
I20260806 04:01:09.450325 3224 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:09.505395 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 126488us
I20260806 04:01:09.463960 3140 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.504664 3209 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.385766 3210 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:09.513049 32652 debug-util.cc:398] Leaking SignalData structure 0x7b0800311020 after lost signal to thread 32530
I20260806 04:01:09.453809 3223 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.386222 3179 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.504215 3231 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.523587 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.527599 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.521687 3049 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:09.523110 3175 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.533618 3152 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.534363 3198 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.533689 3200 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.472223 3206 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.478555 3230 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.493690 3225 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.527580 3170 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.525969 3218 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.527529 3199 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.535388 2794 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.566929 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:09.567543 32652 debug-util.cc:398] Leaking SignalData structure 0x7b0800354120 after lost signal to thread 32659
I20260806 04:01:09.522712 3233 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.535862 3049 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.546268 3160 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.543994 3114 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.532001 3184 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.495537 3219 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.541095 3186 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.541973 3176 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.574379 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.574402 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.574956 3217 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.550210 3174 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.513579 3215 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.553514 3113 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.580657 3049 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:09.585825 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.602092 3224 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.602416 3222 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.507721 3220 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.600971 3232 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.552420 3104 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.551201 3191 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.546922 3234 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.570107 3197 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:09.591863 32478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1459) took 1445 ms (client timeout 1856 ms). Trace:
I20260806 04:01:09.610124 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.614151 32470 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:09.615718 32739 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
W20260806 04:01:09.612710 32478 rpcz_store.cc:269] 0806 04:01:08.823098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:08.824095 (+ 997us) service_pool.cc:227] Handling call
0806 04:01:09.591670 (+767575us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54000488d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:09.591774 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":611293568}
I20260806 04:01:09.587927 3223 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.624567 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.626345 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.585974 3138 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:09.617125 32652 debug-util.cc:398] Leaking SignalData structure 0x7b08003236a0 after lost signal to thread 326
I20260806 04:01:09.625763 32746 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:09.622211 3204 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.625053 3140 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.629509 2794 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.629766 3225 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.630128 3179 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.630473 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.584939 3218 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:09.628757 32652 debug-util.cc:398] Leaking SignalData structure 0x7b0800307860 after lost signal to thread 327
I20260806 04:01:09.615532 3193 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.625262 3183 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.603366 3188 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.604133 3160 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.550869 3054 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.636494 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:09.637185 32652 debug-util.cc:398] Leaking SignalData structure 0x7b08002a50e0 after lost signal to thread 1231
I20260806 04:01:09.636829 3215 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.640434 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.643570 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.643909 3210 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.625049 3203 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.624783 3208 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.644781 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.646827 1137 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:09.624500 3206 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.649350 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.649670 3219 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.614142 3201 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.650875 32745 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:09.651640 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.651677 3224 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.651973 3230 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.652280 3224 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.652486 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.657320 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.657027 3233 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.658375 3220 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.659747 32474 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:09.660804 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.661384 1183 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:09.647049 3184 leader_election.cc:288] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.663053 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.663686 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.664546 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.664907 32682 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:09.665908 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.666589 1192 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:09.665943 3186 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:09.667268 1190 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:09.661087 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.667589 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.668576 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.668767 32745 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:09.670229 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.670955 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.668043 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.671334 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.659755 3234 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.674106 1192 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4f865676f884755987b8bd820489186"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37314
I20260806 04:01:09.677239 1192 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:09.677383 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.677675 1192 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.672055 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.672694 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.673086 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.678831 1137 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:09.679271 32741 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:09.673527 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.680244 3233 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.680550 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:09.680733 3233 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.680284 32743 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:09.682904 3160 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.656735 3224 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.673777 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.686496 32740 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:09.687812 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.689330 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet d24bcc94fb734c95849f4cb89c15ca8e from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:01:09.689571 3218 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.689940 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet d24bcc94fb734c95849f4cb89c15ca8e from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-d24bcc94fb734c95849f4cb89c15ca8e
I20260806 04:01:09.690321 1137 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:09.690979 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.691449 1137 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:09.685032 32478 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:09.691972 32739 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:09.692854 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:09.693246 3219 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.693679 3219 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.689380 32551 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:09.674198 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:09.674803 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.697655 3222 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.697847 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.698155 3222 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.698210 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.698660 3222 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.699548 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.685519 3174 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:09.676450 32682 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:09.697520 3186 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.700363 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.700208 32471 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:09.701879 1137 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:09.691607 32475 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:09.719112 32469 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:09.724402 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.727581 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.729386 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.727480 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.729245 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:09.725790 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 51684.3us
I20260806 04:01:09.729701 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.702643 1192 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.706239 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.694200 3233 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.694051 3219 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.734977 3152 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 04:01:09.743696 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 54535.6us
I20260806 04:01:09.734678 32466 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:09.743217 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.700112 3176 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.702353 3160 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:09.746728 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.747013 1137 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:09.743732 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.709640 3174 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.705173 3188 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:09.784162 1137 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:09.783735 32739 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:09.784572 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.779632 32470 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:09.786494 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.790652 32469 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:09.787071 3198 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.787254 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.787613 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.785615 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.746341 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:09.788185 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 218404us
W20260806 04:01:09.788960 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 44489.4us
I20260806 04:01:09.785965 1192 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.732259 3140 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:09.736334 3212 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.743399 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.793083 32471 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
W20260806 04:01:09.786450 1223 debug-util.cc:398] Leaking SignalData structure 0x7b08002c2860 after lost signal to thread 30649
I20260806 04:01:09.740267 3201 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.743532 3152 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.795948 1137 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:09.704599 32738 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:09.796803 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.790086 3174 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:09.785722 3160 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.801450 3206 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.803295 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.792218 3198 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.795414 3218 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.810743 3218 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.811156 3218 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.803813 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.800797 32551 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:09.805305 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.807196 1137 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:09.797561 3212 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.806170 3225 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:09.800320 3152 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.779953 3186 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:09.814026 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.814884 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.798054 3217 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.801378 3249 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:09.826223 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.808736 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.828025 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.808480 3206 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:09.826928 3249 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:09.831380 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.830474 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.820871 32631 tablet_copy_source_session.cc:205] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:01:09.832296 32745 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:09.799562 3176 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:09.812593 32553 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.237.131:34975 timed out after 1.560s (SENT) [suppressed 4 similar messages]
I20260806 04:01:09.821197 3212 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.813472 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.819674 1137 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:09.820535 3225 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:09.814899 1223 debug-util.cc:398] Leaking SignalData structure 0x7b08002af4a0 after lost signal to thread 32373
I20260806 04:01:09.821938 3231 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:09.816776 3198 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.852396 3054 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.825598 3243 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.853790 3243 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.854342 3243 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.813256 1186 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:01:09.827229 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.807761 3160 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:09.818056 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.832232 3142 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.836459 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:09.836063 3217 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.837600 32477 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:01:09.822154 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.840106 3206 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.840175 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:09.840945 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.842569 32551 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:09.842877 3222 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.862443 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:09.847030 1223 debug-util.cc:398] Leaking SignalData structure 0x7b080003a240 after lost signal to thread 32400
I20260806 04:01:09.847520 3225 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:09.850145 3249 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.807649 3201 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:09.855981 32470 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:09.842336 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.872296 32551 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:09.876442 1187 raft_consensus.cc:2379] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 2: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:09.879272 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.867574 3208 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:09.883862 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.884788 32551 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:09.885313 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.885370 3172 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.902872 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.887225 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 637) took 1041 ms. Trace:
I20260806 04:01:09.879272 3158 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:09.889094 3217 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:09.889321 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:09.896956 1223 debug-util.cc:398] Leaking SignalData structure 0x7b0800389b60 after lost signal to thread 32530
I20260806 04:01:09.860106 3142 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.885813 3222 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:09.906476 3158 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:09.911401 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 105091us
I20260806 04:01:09.888959 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1671) took 1195 ms. Trace:
I20260806 04:01:09.915418 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.913232 32602 rpcz_store.cc:276] 0806 04:01:08.681568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:09.296237 (+614669us) service_pool.cc:227] Handling call
0806 04:01:09.309954 (+ 13717us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:09.542975 (+233021us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:09.543018 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:09.543071 (+ 53us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:09.543106 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:09.862775 (+319669us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b540007dfd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:09.868815 (+ 6040us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:09.876369 (+ 7554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":496056192}
I20260806 04:01:09.918357 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.904395 3172 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.922497 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.922751 32631 tablet_copy_service.cc:140] P dd7344d85a154539b7f8c484f2a76772: Received BeginTabletCopySession request for tablet 12e7971e809846ada28ebf1d6cea4837 from peer d4a238f055a44f459c62c224daa4d541 ({username='slave'} at 127.0.0.1:42618)
I20260806 04:01:09.924247 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.925119 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.911384 3031 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:09.936301 3031 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:09.936914 3031 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:09.937484 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.938771 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.939261 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.925880 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.910012 32738 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.926836 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.923794 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.939900 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.941651 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.926959 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:09.929965 32631 tablet_copy_service.cc:161] P dd7344d85a154539b7f8c484f2a76772: Beginning new tablet copy session on tablet 12e7971e809846ada28ebf1d6cea4837 from peer d4a238f055a44f459c62c224daa4d541 at {username='slave'} at 127.0.0.1:42618: session id = d4a238f055a44f459c62c224daa4d541-12e7971e809846ada28ebf1d6cea4837
I20260806 04:01:09.911618 3158 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.931938 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.932346 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.929414 3172 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.904436 1184 rpcz_store.cc:276] 0806 04:01:08.853548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:08.933797 (+ 80249us) service_pool.cc:227] Handling call
0806 04:01:09.882535 (+948738us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400164754. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:09.887133 (+ 4598us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":246747392}
I20260806 04:01:09.933380 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:09.934628 1223 debug-util.cc:398] Leaking SignalData structure 0x7b08002f3180 after lost signal to thread 32659
I20260806 04:01:09.934751 3098 tablet_copy_client.cc:531] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:01:09.925758 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.909935 3142 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:09.939889 1186 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:09.945325 1137 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:09.882864 32737 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:09.949453 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:09.954125 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:09.947602 1223 debug-util.cc:398] Leaking SignalData structure 0x7b08002f31a0 after lost signal to thread 32660
I20260806 04:01:09.959875 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.960103 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.962757 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:09.965229 3244 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:09.965852 3244 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:09.981631 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.981817 3244 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:09.990026 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 60473.6us
I20260806 04:01:09.955904 32428 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:09.994064 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:09.995651 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:10.000541 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.001111 32604 raft_consensus.cc:3032] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:10.001380 32604 raft_consensus.cc:729] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:10.002815 32682 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:09.997974 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 45514.4us
I20260806 04:01:10.013567 32553 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.022080 32682 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:10.026693 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:10.016366 1223 debug-util.cc:398] Leaking SignalData structure 0x7b080038c540 after lost signal to thread 326
I20260806 04:01:10.031373 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.005386 3114 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:10.012789 3179 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:10.040964 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.042975 32428 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:10.043464 32553 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.044595 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:10.045672 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.045866 3134 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:10.025763 3204 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:10.046288 3134 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:10.051119 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.051913 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.051230 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.052994 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:10.039685 1223 debug-util.cc:398] Leaking SignalData structure 0x7b080038c520 after lost signal to thread 327
I20260806 04:01:10.033252 3176 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:10.061317 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.050645 3134 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:10.061975 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.036527 3179 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:10.037487 3114 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:10.045608 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.036481 3140 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:10.109058 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.032148 32604 consensus_queue.cc:260] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.110706 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.025357 3233 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.051662 3184 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:10.049639 3204 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:10.052248 3201 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:10.119119 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.122539 32604 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.115849 32427 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.110355 3179 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.110936 3114 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:10.132972 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:10.133734 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:10.126106 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:10.110301 3140 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:10.142614 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.122083 3184 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:10.125636 3204 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:10.138504 1223 debug-util.cc:398] Leaking SignalData structure 0x7b080038c9a0 after lost signal to thread 1231
I20260806 04:01:10.149192 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:10.167200 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:10.173607 32683 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.141058 3140 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:10.136606 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:10.181461 1223 debug-util.cc:398] Leaking SignalData structure 0x7b080038c820 after lost signal to thread 1233
I20260806 04:01:10.172561 3184 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:10.199930 3201 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:10.232836 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.168700 3123 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:01:10.233106 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 188573us
I20260806 04:01:10.190860 3230 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.194602 32683 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:10.257390 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 62886.1us
I20260806 04:01:10.258668 32683 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.260147 32683 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:10.256170 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:10.169654 3218 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.255618 3123 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:10.258396 3223 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.276435 3201 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:10.314090 32631 tablet_copy_source_session.cc:205] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:01:10.282025 3123 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:10.282793 3188 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:10.346989 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.356791 3257 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:10.357506 3257 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:10.359501 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 325017us
I20260806 04:01:10.329401 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.345448 3233 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.367375 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:10.366336 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 79005.3us
W20260806 04:01:10.369784 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 152862us
I20260806 04:01:10.344915 3188 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:10.437107 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 177027us
W20260806 04:01:10.439574 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 191674us
I20260806 04:01:10.440521 32553 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.445999 3278 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:01:10.448122 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 183867us
I20260806 04:01:10.381201 3188 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:10.456421 32683 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:10.452173 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 83715us
I20260806 04:01:10.472223 32683 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:10.452661 3224 consensus_meta.cc:287] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.766s user 0.015s sys 0.000s
I20260806 04:01:10.476315 3144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24bcc94fb734c95849f4cb89c15ca8e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 04:01:10.470737 3049 consensus_meta.cc:287] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.857s user 0.010s sys 0.006s
I20260806 04:01:10.469122 3201 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 04:01:10.490686 3152 consensus_meta.cc:287] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.662s user 0.020s sys 0.001s
I20260806 04:01:10.480502 3224 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.494982 1190 raft_consensus.cc:1226] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.495978 3224 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:10.496219 3172 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.547s user 0.004s sys 0.014s
I20260806 04:01:10.496603 3172 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.497633 3184 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:10.498317 32604 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:10.498495 3172 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:10.499282 3257 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.504407 32473 raft_consensus.cc:2379] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:10.508028 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 67591.4us
I20260806 04:01:10.508774 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:10.491194 3152 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:10.509600 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.510615 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.510247 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.509793 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.512279 32466 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.486085 3201 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:10.492591 3231 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:01:10.508126 3198 consensus_meta.cc:287] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.654s user 0.009s sys 0.009s
I20260806 04:01:10.504885 3184 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.508903 3257 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.505760 3275 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:10.514945 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.523470 1136 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.522708 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.510288 3218 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:10.525656 32553 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.485930 3049 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.519272 3201 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.537372 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:10.538849 3225 consensus_meta.cc:287] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.634s user 0.005s sys 0.012s
I20260806 04:01:10.539323 3225 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:10.540313 3212 consensus_meta.cc:287] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.685s user 0.006s sys 0.010s
I20260806 04:01:10.540745 3212 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.541371 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.542608 3212 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:10.543462 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.546173 3261 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:10.544721 1136 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.553002 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.558256 32748 raft_consensus.cc:1226] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:10.534458 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.527879 3281 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:10.563231 1136 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:10.551442 3219 consensus_meta.cc:287] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.766s user 0.013s sys 0.003s
I20260806 04:01:10.541371 3225 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:10.554744 3244 consensus_meta.cc:287] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.557s user 0.004s sys 0.012s
I20260806 04:01:10.555444 3172 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.564769 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:10.530690 3275 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:10.521553 3198 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.598268 32553 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.599517 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:10.566975 3160 consensus_meta.cc:287] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.691s user 0.009s sys 0.005s
W20260806 04:01:10.605397 3158 consensus_meta.cc:287] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.647s user 0.012s sys 0.004s
I20260806 04:01:10.605868 3158 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.607784 3158 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:10.608117 3142 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.623s user 0.013s sys 0.006s
I20260806 04:01:10.608484 3142 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.609863 3142 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:10.569713 3281 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:10.613941 32466 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:10.567539 3261 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:10.616868 1186 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.616773 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.618578 1137 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:10.616189 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:10.600273 3225 raft_consensus.cc:2726] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:10.567019 3204 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.624270 32749 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:10.628978 3049 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:10.629820 3031 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.692s user 0.016s sys 0.002s
I20260806 04:01:10.630280 3031 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:10.631158 3134 consensus_meta.cc:287] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.001s
I20260806 04:01:10.631569 3134 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.633426 3134 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:10.634078 3204 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:10.630585 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.603062 3275 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.641177 3198 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.642722 3188 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:01:10.643390 3243 consensus_meta.cc:287] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.788s user 0.008s sys 0.014s
I20260806 04:01:10.644904 3186 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:10.638543 3257 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:10.617431 3281 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:10.653498 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:10.655232 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:10.658334 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:10.659519 3174 consensus_meta.cc:287] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.839s user 0.006s sys 0.013s
I20260806 04:01:10.678216 3174 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.679368 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 656) took 1013 ms. Trace:
I20260806 04:01:10.682327 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.682825 3174 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.620610 3142 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.593043 3219 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.699254 3188 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:10.639753 32427 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.667711 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:10.635073 3218 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:10.664140 3257 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:10.703392 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:10.686297 32604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:01:10.683640 1188 raft_consensus.cc:2379] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:10.679764 1190 rpcz_store.cc:276] 0806 04:01:09.670488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:09.670640 (+ 152us) service_pool.cc:227] Handling call
0806 04:01:09.670966 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:10.495358 (+824392us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:10.679154 (+183796us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b540078f5d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:10.679306 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1894658304}
I20260806 04:01:10.695873 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.698045 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.715334 3098 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:01:10.715520 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:10.716923 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.704373 32604 raft_consensus.cc:595] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Received request to transfer leadership to TS d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.614087 3160 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.702167 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:10.700661 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.703589 1189 raft_consensus.cc:1226] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.597720 3244 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:10.706450 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 66778.7us
I20260806 04:01:10.707226 3218 raft_consensus.cc:686] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:10.708163 3257 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.711114 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.710677 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:10.712675 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.721237 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.720655 1183 raft_consensus.cc:1226] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:10.722014 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.724906 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.711632 3140 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.727952 32427 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.689486 3031 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:10.643836 3114 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:10.723641 32682 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.734894 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.736033 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:10.736634 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.713801 3287 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.703958 3152 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.753835 3218 consensus_queue.cc:237] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.666129 3243 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.758797 3243 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.763500 3188 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:10.740290 1185 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
W20260806 04:01:10.766124 3179 consensus_meta.cc:287] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.623s user 0.006s sys 0.006s
I20260806 04:01:10.766120 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 690) took 1068 ms. Trace:
I20260806 04:01:10.766696 3179 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.739184 32610 raft_consensus.cc:1226] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.767488 1185 rpcz_store.cc:276] 0806 04:01:09.702532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:09.723544 (+ 21012us) service_pool.cc:227] Handling call
0806 04:01:09.723878 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:10.765213 (+1041335us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:10.765845 (+ 632us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54006f0ed4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:10.766022 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2336931200}
I20260806 04:01:10.768909 32737 raft_consensus.cc:1226] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.741786 1137 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.744721 3230 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:10.746317 1182 raft_consensus.cc:2379] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:10.740123 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.779081 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.780402 32608 raft_consensus.cc:1226] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.781008 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.781708 32610 raft_consensus.cc:2379] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 2: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:10.747225 3114 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.748919 32682 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.751451 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 691) took 1048 ms. Trace:
I20260806 04:01:10.752246 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260806 04:01:10.752359 3193 consensus_queue.cc:1046] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:10.754921 3160 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.748536 3222 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 04:01:10.770398 32738 consensus_meta.cc:287] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.787s user 0.016s sys 0.000s
I20260806 04:01:10.732772 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.710659 3123 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:10.778298 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.785550 32427 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.813710 32605 raft_consensus.cc:1226] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:10.788661 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.784940 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:10.787665 3212 raft_consensus.cc:2726] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:10.783867 1183 rpcz_store.cc:276] 0806 04:01:09.702904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:09.708803 (+ 5899us) service_pool.cc:227] Handling call
0806 04:01:10.720152 (+1011349us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:10.740573 (+ 20421us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:10.751192 (+ 10619us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b54005bac50. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:10.751357 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2140868480}
I20260806 04:01:10.816542 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:10.793210 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.779116 32602 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:01:10.818109 3123 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.794545 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.796293 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.797930 3193 raft_consensus.cc:3032] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:10.820833 1137 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.821301 3193 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:10.821955 3212 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:10.787952 3223 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:10.807521 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.822846 3193 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.833348 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.833755 3230 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.834477 3222 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.833345 3193 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:01:10.829324 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 43856.1us
I20260806 04:01:10.830286 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.836066 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.836558 32748 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:10.844383 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.814256 32738 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:10.850008 3140 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:10.850759 32745 raft_consensus.cc:1226] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.851362 32748 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:10.862457 3179 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:10.865108 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:10.788105 3219 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.866914 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:10.867919 3287 consensus_queue.cc:1046] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:10.870296 32683 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:10.870929 3287 raft_consensus.cc:3032] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:10.871476 3287 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:10.872654 3244 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:10.873302 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:10.875465 3283 consensus_queue.cc:1046] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:10.879382 32745 raft_consensus.cc:2379] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:10.872630 3287 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.884392 3230 consensus_queue.cc:1046] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:10.885149 3283 raft_consensus.cc:3032] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:10.885574 3287 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:10.878750 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 144997us
I20260806 04:01:10.850740 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33182 (request call id 266) took 1030 ms. Trace:
I20260806 04:01:10.885820 3114 raft_consensus.cc:3032] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:10.894999 3114 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:10.896133 3114 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.897574 3114 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:10.886868 3283 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:10.899243 3283 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.890437 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.892992 1182 raft_consensus.cc:1226] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.905948 3201 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:10.889765 32551 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:10.913928 3031 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:10.914430 3031 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:10.914996 3031 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:10.920938 3249 consensus_queue.cc:1046] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:10.878804 3179 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:10.923498 3230 raft_consensus.cc:3032] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:10.924099 3230 raft_consensus.cc:729] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:10.892593 3281 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:10.926232 3152 raft_consensus.cc:979] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772: : Instructing follower d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:01:10.925374 3230 consensus_queue.cc:260] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.932809 1136 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:10.928753 3152 raft_consensus.cc:1067] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Signalling peer d4a238f055a44f459c62c224daa4d541 to start an election
I20260806 04:01:10.936661 32682 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.937947 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.942531 3158 consensus_queue.cc:1046] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:10.944272 3212 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:10.945475 3123 consensus_queue.cc:1046] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:10.945926 3292 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:10.944367 1182 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.939216 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.949456 1136 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:10.939427 32428 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:10.950302 32682 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.951221 32609 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:10.951519 3176 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.946753 3274 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.923508 3283 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:10.955694 32682 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.958369 32682 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.960737 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.964768 3188 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:10.966502 3158 raft_consensus.cc:3032] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:10.967052 3158 raft_consensus.cc:729] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:10.968657 3158 consensus_queue.cc:260] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:10.970614 3134 raft_consensus.cc:1050] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:01:10.971920 3200 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:10.972688 3200 raft_consensus.cc:686] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:10.974007 3200 consensus_queue.cc:237] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:10.977041 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.977576 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.989912 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.990561 32605 raft_consensus.cc:1226] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:10.983707 32682 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:10.976918 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.993525 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.980576 1189 raft_consensus.cc:1226] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.991567 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.995596 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:10.995816 3230 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.996865 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.981644 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:10.996780 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:10.996745 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.995611 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.948505 3223 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.980849 1185 raft_consensus.cc:1226] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:10.994676 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:10.891222 32738 rpcz_store.cc:276] 0806 04:01:09.819822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:09.839814 (+ 19992us) service_pool.cc:227] Handling call
0806 04:01:10.850428 (+1010614us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b540011dad4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:10.850625 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91428608}
I20260806 04:01:10.976735 3176 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.000240 1187 raft_consensus.cc:1226] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:11.001029 32682 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.002214 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:11.002517 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:11.002646 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:11.003161 3158 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:10.990042 3031 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.004005 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.949687 3257 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.003494 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:11.006453 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:11.003159 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:11.008210 3210 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:10.993016 3188 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.032104 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:11.017639 32740 raft_consensus.cc:1226] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:10.926872 3294 consensus_queue.cc:1046] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260806 04:01:11.015785 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:10.977813 3140 raft_consensus.cc:3032] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:10.992461 3219 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:10.974470 3134 consensus_queue.cc:237] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.048761 32614 raft_consensus.cc:1226] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.020223 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:11.021301 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:11.024922 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:11.049572 32610 raft_consensus.cc:2379] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:11.014540 32682 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.034096 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:11.038628 3172 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:11.039371 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:11.039849 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:11.044425 32609 raft_consensus.cc:2379] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:11.044440 32603 raft_consensus.cc:1226] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:11.021201 3292 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.020917 3152 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:11.097514 3152 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.032096 3178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e7971e809846ada28ebf1d6cea4837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:01:11.089172 3144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37314 (request call id 628) took 3624 ms. Trace:
I20260806 04:01:11.086489 3275 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.044721 3140 raft_consensus.cc:729] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.095561 3172 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.096976 32615 raft_consensus.cc:1226] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.104696 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:11.105090 32608 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:11.107225 1136 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.107784 32427 leader_election.cc:302] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.109102 32467 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.110661 3262 consensus_queue.cc:1046] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:11.090935 32602 raft_consensus.cc:2379] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:11.123333 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.084806 3201 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:11.120007 1136 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.121368 32604 raft_consensus.cc:1226] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.120419 3243 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 04:01:11.113965 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 99503.5us
I20260806 04:01:11.091558 3219 raft_consensus.cc:686] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:11.128899 32682 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.129742 1136 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.134044 1136 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.131769 32427 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.127758 32612 raft_consensus.cc:1226] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.136976 3163 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.137883 3253 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.138864 3198 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.140056 3285 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.142227 32427 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.144519 32682 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.145323 32427 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.117422 3292 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.147802 32682 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.150956 32427 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.155367 32427 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.129559 3294 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.166211 1136 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.153007 3201 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.155306 3278 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.154368 3203 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.152900 3231 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.156487 3049 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.174396 1136 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.128659 3160 consensus_queue.cc:1046] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260806 04:01:11.161613 3253 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:11.175849 3274 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.153252 3152 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:11.166800 3281 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:11.171178 3243 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.172677 3208 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.184752 3215 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.156400 3290 consensus_queue.cc:1046] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260806 04:01:11.173261 3285 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.170044 3123 consensus_queue.cc:1046] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260806 04:01:11.171201 3262 consensus_queue.cc:1046] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20260806 04:01:11.204053 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 45768.2us
I20260806 04:01:11.200897 32740 raft_consensus.cc:1226] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.202512 1183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
from {username='slave'} at 127.0.0.1:37314
I20260806 04:01:11.209843 1184 raft_consensus.cc:1226] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.210024 1183 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:11.210445 1183 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:01:11.189390 3295 log.cc:924] Time spent T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260806 04:01:11.202509 3219 consensus_queue.cc:237] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.193069 3031 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:11.223646 32467 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:11.200141 3257 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:11.225188 32683 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:11.227319 3172 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:11.233016 3249 raft_consensus.cc:3032] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.233505 3249 raft_consensus.cc:729] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:11.234572 3142 raft_consensus.cc:3032] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.234962 3142 raft_consensus.cc:729] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.235930 3142 consensus_queue.cc:260] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.107810 3144 rpcz_store.cc:276] 0806 04:01:07.465228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:07.469325 (+ 4097us) service_pool.cc:227] Handling call
0806 04:01:07.997710 (+528385us) ts_tablet_manager.cc:928] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
0806 04:01:10.033281 (+2035571us) tablet_copy_client.cc:361] Tablet copy session begun
0806 04:01:10.758843 (+725562us) tablet_metadata.cc:641] Metadata flushed
0806 04:01:10.759780 (+ 937us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 04:01:11.078496 (+318716us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 04:01:11.088416 (+ 9920us) tablet_replica.cc:205] Creating consensus instance
0806 04:01:11.088900 (+ 484us) ts_tablet_manager.cc:983] Replica registered.
0806 04:01:11.089051 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":1032,"tablet-copy.queue_time_us":511090,"thread_start_us":2048618,"threads_started":2}
I20260806 04:01:11.237237 3142 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.237565 3144 tablet_copy_client.cc:791] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
I20260806 04:01:11.238121 3144 tablet_copy_client.cc:655] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:01:11.174603 3140 consensus_queue.cc:260] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.253742 3275 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:11.255203 3186 consensus_queue.cc:1046] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:11.224072 3285 consensus_queue.cc:1046] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 04:01:11.262811 3261 raft_consensus.cc:3032] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:11.263231 3261 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:11.264204 3261 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.249655 3140 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:01:11.268712 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 113416us
W20260806 04:01:11.270097 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 45007.2us
I20260806 04:01:11.273365 3303 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:11.274335 3186 raft_consensus.cc:3032] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.285760 3186 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:11.285315 3303 raft_consensus.cc:686] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:11.287477 32427 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:11.287204 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 99478.7us
I20260806 04:01:11.295436 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:11.297621 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 69291.5us
I20260806 04:01:11.298066 32427 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.298988 32682 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:11.302619 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 50962.6us
I20260806 04:01:11.316531 1188 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:11.318065 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.295225 3303 consensus_queue.cc:237] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:11.317356 32682 leader_election.cc:302] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.333354 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.303701 3054 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.338256 32553 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.338735 32427 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.303957 3176 consensus_queue.cc:1046] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260806 04:01:11.348636 3144 tablet_copy_client.cc:531] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:01:11.320123 3253 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:11.350368 3283 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:11.327867 3294 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.335640 3186 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.363293 3186 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.333695 3231 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.333439 3223 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.336530 3049 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:11.368386 3152 raft_consensus.cc:3032] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.368960 3152 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.364907 3274 consensus_queue.cc:1046] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260806 04:01:11.373229 3301 raft_consensus.cc:3032] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:11.373730 3301 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:11.373697 3152 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.375789 3301 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.379411 3249 consensus_queue.cc:260] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.366047 3314 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.368165 3280 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.381937 32553 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.381246 3249 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.381835 3049 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.391732 3049 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:11.392721 3257 raft_consensus.cc:3032] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.393338 3257 raft_consensus.cc:729] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.393705 1183 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.394593 3257 consensus_queue.cc:260] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.395774 3261 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:11.396965 3257 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:01:11.398372 3114 consensus_meta.cc:287] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.500s user 0.007s sys 0.003s
I20260806 04:01:11.398281 1183 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:11.340560 3287 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:11.400436 3287 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:11.403033 3172 raft_consensus.cc:3032] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:11.403343 3287 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:11.403491 3172 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.408356 1184 raft_consensus.cc:1226] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.410267 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:11.410848 32748 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.415355 32427 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.421876 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:11.424822 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:11.426524 3172 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.427429 3144 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:01:11.428292 3172 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:11.431640 3318 consensus_queue.cc:1046] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:11.432763 3203 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:11.422132 32477 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:11.430706 32326 catalog_manager.cc:5495] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 2 to 3, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:11.439476 3200 consensus_queue.cc:1046] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:11.442178 3200 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:11.442701 3200 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:11.417308 3140 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.453274 3259 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:11.462042 3317 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:11.462694 3317 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:11.462838 3305 raft_consensus.cc:3032] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.463404 3317 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.470448 3305 raft_consensus.cc:729] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.470711 3152 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.471791 3305 consensus_queue.cc:260] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.473747 3305 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.451437 32748 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:11.477201 1137 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:11.489316 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 65049.6us
W20260806 04:01:11.490156 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 68342.2us
I20260806 04:01:11.493683 32427 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.444018 3200 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.498793 1184 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:01:11.500944 3200 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:11.503176 3317 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.505597 3318 consensus_queue.cc:1046] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:11.490540 3230 consensus_queue.cc:1046] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 6.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260806 04:01:11.509488 3294 raft_consensus.cc:3032] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:11.509974 3294 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:11.493276 3208 raft_consensus.cc:2781] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:11.512754 3208 raft_consensus.cc:686] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:11.420153 3114 consensus_queue.cc:1046] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260806 04:01:11.514240 3208 consensus_queue.cc:237] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.521284 32605 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:11.521020 32322 catalog_manager.cc:5495] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:11.521831 3278 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.511185 3294 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.527896 32322 catalog_manager.cc:5495] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:11.529278 32553 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.524573 3142 raft_consensus.cc:3032] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.527829 3134 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260806 04:01:11.518661 3257 consensus_queue.cc:1046] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260806 04:01:11.536710 32553 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.524214 3317 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:11.476975 3203 consensus_queue.cc:1046] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260806 04:01:11.531148 3142 raft_consensus.cc:729] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.548794 3284 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:11.557199 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:11.564291 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 244053us
I20260806 04:01:11.566764 32682 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:11.538194 3296 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.571115 3301 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:11.572871 32682 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.541270 3223 raft_consensus.cc:3032] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.579087 3223 raft_consensus.cc:729] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:11.567641 3294 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:11.576709 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.578400 3031 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:01:11.580823 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 44201us
I20260806 04:01:11.582544 3142 consensus_queue.cc:260] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.582736 3223 consensus_queue.cc:260] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.544060 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:11.585475 3140 raft_consensus.cc:3032] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:11.590641 3285 raft_consensus.cc:3032] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.574879 3321 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:11.616788 3321 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:11.617476 3321 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.601202 32682 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.602603 32553 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.603941 32427 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.603971 32603 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.595685 3283 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:11.604701 3188 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:11.599992 3274 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.622977 32615 raft_consensus.cc:1226] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:11.607221 3140 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.618412 3285 raft_consensus.cc:729] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:11.623003 3114 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.628393 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.625861 3176 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.629758 3152 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.655336 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:11.664317 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:11.642702 3223 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.662855 32603 raft_consensus.cc:3032] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:11.668378 32603 raft_consensus.cc:729] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.640811 3249 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:11.668963 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.669275 3249 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.673460 32748 raft_consensus.cc:1261] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:11.636560 3257 consensus_queue.cc:1046] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260806 04:01:11.634796 3243 consensus_queue.cc:1046] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260806 04:01:11.639843 3274 consensus_queue.cc:1046] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260806 04:01:11.667266 3031 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:11.682924 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.685336 32605 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:11.682652 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:11.657426 3303 consensus_queue.cc:1046] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260806 04:01:11.689476 3274 raft_consensus.cc:3032] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:11.690016 3274 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:11.690908 3274 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.679661 32603 consensus_queue.cc:260] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:11.720883 32603 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:11.739310 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 56467.4us
I20260806 04:01:11.739950 32553 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.697081 3314 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.746419 32553 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.677450 3283 consensus_queue.cc:1046] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 04:01:11.734354 3134 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.748142 3158 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.667737 3285 consensus_queue.cc:260] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.773684 3249 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:11.720520 3244 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260806 04:01:11.745332 3280 raft_consensus.cc:3032] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.782037 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:11.784125 32603 raft_consensus.cc:1261] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:11.785615 32475 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.786970 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:11.789258 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:11.747967 3031 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:11.661477 3319 consensus_queue.cc:1046] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.101s
I20260806 04:01:11.799693 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:11.801370 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:11.802183 32553 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:11.803285 32553 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:11.796495 3280 raft_consensus.cc:729] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:11.799005 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:11.779769 3274 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:11.804335 32553 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:11.806200 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:11.791491 3054 raft_consensus.cc:3032] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.812240 32473 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:01:11.816326 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:11.811513 32471 raft_consensus.cc:1226] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:11.816888 32478 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
W20260806 04:01:11.857781 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 53514.2us
I20260806 04:01:11.817148 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:11.821511 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:11.858635 32553 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:11.831660 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:11.806900 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:11.837224 3259 raft_consensus.cc:3032] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:11.809957 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:11.822175 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:11.843556 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:11.843580 3140 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.811473 32321 catalog_manager.cc:5495] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 04:01:11.885632 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 205105us
W20260806 04:01:11.822899 3322 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260806 04:01:11.810601 3134 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.889729 32682 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:11.827219 3253 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:11.847110 3278 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.821784 3319 raft_consensus.cc:3032] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.814330 32737 raft_consensus.cc:1226] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:11.850737 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:11.853282 3188 raft_consensus.cc:3032] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:11.868255 3257 raft_consensus.cc:2726] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:11.855523 3285 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.867262 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:11.868985 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 215892us
I20260806 04:01:11.874169 32467 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:01:11.804745 3172 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.889133 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:11.820616 3142 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.904644 32466 raft_consensus.cc:1226] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:11.905326 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:11.856591 3054 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:11.859882 32553 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:11.917637 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 57841.6us
I20260806 04:01:11.909580 32741 raft_consensus.cc:1226] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:11.868736 3259 raft_consensus.cc:729] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:11.883260 3031 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.915405 32746 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 1. Current term is 2. Ops: []
W20260806 04:01:11.916560 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 47371.7us
I20260806 04:01:11.901155 3179 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:11.903789 3319 raft_consensus.cc:729] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:11.798422 3325 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
I20260806 04:01:11.912043 32740 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:11.907495 3188 raft_consensus.cc:729] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:11.944963 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:11.821086 3280 consensus_queue.cc:260] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.929150 3257 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:11.941108 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:11.954891 1182 raft_consensus.cc:2379] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 2: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:11.958845 3098 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:01:11.940017 3176 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260806 04:01:11.987567 3249 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:11.938043 3140 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.002302 3054 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.009218 32469 raft_consensus.cc:1226] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:12.012555 3319 consensus_queue.cc:260] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.015321 3280 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:11.962462 3259 consensus_queue.cc:260] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:11.963738 3172 consensus_queue.cc:1046] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:12.017305 3172 raft_consensus.cc:3032] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:12.017810 3172 raft_consensus.cc:729] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:11.965298 32553 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:12.020871 3142 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 04:01:12.028745 3200 consensus_queue.cc:1159] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d4a238f055a44f459c62c224daa4d541" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20260806 04:01:12.033459 32612 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:12.032949 3188 consensus_queue.cc:260] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.043179 32748 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 04:01:12.066218 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 100963us
I20260806 04:01:12.074676 32745 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:12.019058 3172 consensus_queue.cc:260] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:12.092497 3098 tablet_bootstrap.cc:492] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:01:12.096058 3172 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:12.096554 3098 ts_tablet_manager.cc:1397] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 1.383s user 0.030s sys 0.033s
W20260806 04:01:12.107247 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 182775us
I20260806 04:01:12.081856 3259 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.081132 3054 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:12.096774 3197 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:12.105667 3244 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260806 04:01:12.113716 32470 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
W20260806 04:01:12.114354 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 67120.8us
I20260806 04:01:12.115973 3321 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.105844 3257 consensus_queue.cc:1046] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260806 04:01:12.127027 32553 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:12.136780 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:12.140965 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:12.142935 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 219122us
I20260806 04:01:12.150525 32428 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:12.087322 3031 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260806 04:01:12.153249 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:12.142217 3305 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:12.142407 3134 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:12.158381 3230 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:12.174939 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:12.158397 3314 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:12.127898 3197 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.190062 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:12.162245 3257 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 04:01:12.213440 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 45221.2us
W20260806 04:01:12.183898 3178 consensus_meta.cc:287] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.540s user 0.012s sys 0.000s
I20260806 04:01:12.180370 3230 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:12.229676 32745 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:12.214920 3158 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:12.177399 3318 consensus_queue.cc:1046] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260806 04:01:12.227197 3134 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:12.235240 3230 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:12.240257 3283 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.275099 3283 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.270012 32605 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e"
dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:51186
I20260806 04:01:12.277997 32605 raft_consensus.cc:595] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:12.270462 3098 raft_consensus.cc:348] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:12.273432 32738 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:12.295397 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:12.346119 32748 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:12.296329 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:12.304471 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260806 04:01:12.304268 32743 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
W20260806 04:01:12.292673 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 52704.4us
I20260806 04:01:12.349303 32428 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:12.301815 3348 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260806 04:01:12.351730 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 142493us
I20260806 04:01:12.352849 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:12.353674 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:12.285339 3283 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.284973 3098 raft_consensus.cc:729] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:01:12.291419 3134 raft_consensus.cc:686] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:12.302584 3178 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37314 (request call id 632) took 3570 ms. Trace:
I20260806 04:01:12.366235 3188 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:12.369218 3098 consensus_queue.cc:260] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:12.347275 3280 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.371111 3280 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.349145 3274 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:12.357805 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:12.301281 3197 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:12.294536 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:12.375898 32743 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:12.379302 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 167263us
I20260806 04:01:12.351929 3321 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:12.372457 3274 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.382520 32466 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:12.356448 3285 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:12.383083 32553 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:12.381664 1184 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:12.386399 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:12.390825 32477 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:12.400470 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:12.401721 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:12.413872 3098 ts_tablet_manager.cc:1428] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.317s user 0.005s sys 0.008s
I20260806 04:01:12.407909 3031 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:12.415696 3219 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:12.417726 3134 consensus_queue.cc:237] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.421722 32766 tablet_copy_service.cc:342] P f7c202b562ce482db283463d7f02e136: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-e4533f141e644c719df87540a1ca54e8 received from {username='slave'} at 127.0.0.1:33182
I20260806 04:01:12.418987 3280 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.403403 3158 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:12.416915 3319 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:12.428206 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:12.388003 3152 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:12.391644 3285 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:12.449697 32766 tablet_copy_service.cc:433] P f7c202b562ce482db283463d7f02e136: ending tablet copy session d4a238f055a44f459c62c224daa4d541-e4533f141e644c719df87540a1ca54e8 on tablet e4533f141e644c719df87540a1ca54e8 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:01:12.423991 32602 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:12.451150 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:12.457907 3152 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:12.459790 3285 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:12.462437 3215 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:12.408320 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:12.476946 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 75303.5us
I20260806 04:01:12.481387 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:12.481984 32466 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:12.483371 32683 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.482502 3031 raft_consensus.cc:686] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:12.477694 3215 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:12.499585 3274 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.487651 3201 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:12.373476 3178 rpcz_store.cc:276] 0806 04:01:08.732574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:08.733837 (+ 1263us) service_pool.cc:227] Handling call
0806 04:01:09.223083 (+489246us) ts_tablet_manager.cc:928] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
0806 04:01:10.955340 (+1732257us) tablet_copy_client.cc:361] Tablet copy session begun
0806 04:01:11.596069 (+640729us) tablet_metadata.cc:641] Metadata flushed
0806 04:01:11.624176 (+ 28107us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 04:01:12.225152 (+600976us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 04:01:12.261554 (+ 36402us) tablet_replica.cc:205] Creating consensus instance
0806 04:01:12.302448 (+ 40894us) ts_tablet_manager.cc:983] Replica registered.
0806 04:01:12.302515 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":2298,"tablet-copy.queue_time_us":36927,"thread_start_us":52220,"threads_started":2}
I20260806 04:01:12.488801 3356 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.489279 3353 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:12.485318 32738 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:12.510494 3031 consensus_queue.cc:237] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.498750 3215 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:12.529290 3353 raft_consensus.cc:686] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:12.530344 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42618 (request call id 404) took 1035 ms. Trace:
I20260806 04:01:12.530790 32602 rpcz_store.cc:276] 0806 04:01:11.494829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:11.515583 (+ 20754us) service_pool.cc:227] Handling call
0806 04:01:12.530063 (+1014480us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54000cb554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:12.530257 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":854675328}
I20260806 04:01:12.535342 32615 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 04:01:12.543912 32428 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.545063 3203 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:12.545480 3203 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:12.545800 3203 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:12.548334 3305 raft_consensus.cc:3032] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:12.548804 3305 raft_consensus.cc:729] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:12.505484 3201 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:12.562417 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 83288.2us
I20260806 04:01:12.517293 3178 tablet_copy_client.cc:791] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 0 data blocks...
W20260806 04:01:12.563953 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 47755.2us
I20260806 04:01:12.499209 3314 consensus_queue.cc:1046] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260806 04:01:12.562093 32469 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:12.545730 3244 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.564245 3201 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:12.568917 3142 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.530396 3353 consensus_queue.cc:237] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.549889 3305 consensus_queue.cc:260] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:12.602474 32682 consensus_peers.cc:487] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 -> Peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Couldn't send request to peer d4a238f055a44f459c62c224daa4d541. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12e7971e809846ada28ebf1d6cea4837. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:01:12.581708 3178 tablet_copy_client.cc:655] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: tablet copy: Starting download of 1 WAL segments...
I20260806 04:01:12.620042 3054 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.622504 3344 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.617305 3359 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.515247 3356 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.624221 3359 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.626072 3356 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.615453 3142 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.633183 3203 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.634191 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:12.635001 3142 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.598446 3244 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.610178 3158 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:12.637670 3158 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.9, Last appended by leader: 9, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.638422 3244 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:12.641613 3305 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.648866 1184 raft_consensus.cc:1203] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260806 04:01:12.651253 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 109500us
I20260806 04:01:12.654562 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:12.622675 3172 consensus_meta.cc:287] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.526s user 0.004s sys 0.008s
I20260806 04:01:12.665096 1187 raft_consensus.cc:2379] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 2: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:12.664412 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:12.627712 3359 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:12.671536 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:12.673800 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:12.674695 32737 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:12.676272 3208 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:12.676791 32749 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:12.623114 3344 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.676632 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:12.677647 3230 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.678445 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:12.678381 32473 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:12.679072 32477 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:12.620867 3054 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.680346 32553 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.681141 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:12.666558 32745 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:12.681769 1190 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:12.686970 1137 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:12.695473 1137 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:12.697104 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:12.701761 32553 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.697775 1192 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:12.704209 3031 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:12.704599 3284 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:12.705211 3284 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:12.705685 3284 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:12.706851 3031 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:12.708503 3031 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.711688 3054 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.712462 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:12.713088 32749 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:12.713605 32612 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 04:01:12.714051 3344 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.714879 32428 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.700518 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:12.715818 3054 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:12.700246 3244 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:12.716480 3134 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:12.716660 3244 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:12.717150 3134 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:12.717218 3244 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.717722 3134 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.721637 3031 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.722208 3285 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.716391 3054 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:12.716178 32610 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:12.723091 3178 tablet_copy_client.cc:531] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:01:12.728924 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:12.729660 3201 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.744326 32746 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:12.744690 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:12.745468 32604 raft_consensus.cc:2379] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:12.713323 3203 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.748484 32477 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:12.750025 3285 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.730723 3188 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:12.751041 3188 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.752779 3031 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.737061 3054 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:12.743829 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:12.758036 32683 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.758414 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:12.755321 3152 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:12.763437 3176 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.770478 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:12.771199 32473 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:12.774435 32469 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:12.774554 32683 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:12.775938 32683 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.770733 3134 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.779937 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:12.780565 32748 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.781630 3303 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:12.753758 3188 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.786458 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:12.787017 32614 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:12.787297 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:12.787842 32749 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:01:12.790866 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.777244 3341 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:12.716955 3215 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:12.786687 3348 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260806 04:01:12.784297 3231 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:12.783987 3303 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:12.763967 3176 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.788697 3152 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:12.803872 3231 raft_consensus.cc:686] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:12.805572 3303 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.793871 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:12.810745 1182 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:12.812060 3134 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.793649 3201 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.801242 3274 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:12.798552 3230 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.818991 32748 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:12.804462 3341 raft_consensus.cc:686] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:12.838802 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:12.805855 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:12.843053 32553 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.840613 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:12.844190 32741 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.841730 1189 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.844941 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:12.847503 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:12.848603 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:12.849344 1185 raft_consensus.cc:2379] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 2: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:12.798717 3054 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.850288 3178 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Bootstrap starting.
I20260806 04:01:12.851790 1192 raft_consensus.cc:2379] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:12.858580 3054 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.858902 3134 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:12.859109 3285 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:12.859432 3134 raft_consensus.cc:686] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:12.859678 3285 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.855827 32749 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:12.867621 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:12.868564 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:12.851625 1183 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:12.869184 1190 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.870852 32428 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.871696 3054 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:12.872438 3054 raft_consensus.cc:686] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:12.873775 3054 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:12.877079 3384 log.cc:924] Time spent T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260806 04:01:12.878865 1190 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
W20260806 04:01:12.881222 3383 log.cc:924] Time spent T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260806 04:01:12.839677 3274 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:12.883153 3274 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:12.883971 3144 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:01:12.884905 3144 tablet_bootstrap.cc:492] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:01:12.885916 3144 ts_tablet_manager.cc:1397] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 1.459s user 0.029s sys 0.031s
I20260806 04:01:12.886919 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:12.890189 32749 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:12.860347 3134 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.901759 32749 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:12.903087 32428 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:12.896797 3203 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.909406 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:12.863720 1184 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:12.903978 3283 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:12.932466 3353 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:12.939492 3386 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
W20260806 04:01:12.945115 3389 log.cc:924] Time spent T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260806 04:01:12.932493 3174 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.945851 3341 consensus_queue.cc:237] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:12.947975 3285 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:12.907510 3319 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260806 04:01:12.961190 32741 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:12.907650 3367 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.964463 1189 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:12.965920 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:12.983903 32469 raft_consensus.cc:1261] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:01:12.985311 32749 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:12.985616 32551 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:12.975733 3283 raft_consensus.cc:686] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:12.988941 3283 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.996402 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:12.955559 3303 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.995898 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:12.999258 3176 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:12.974011 3353 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.994534 3140 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:13.005121 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:13.007279 1189 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:13.009678 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:13.009881 32473 raft_consensus.cc:2354] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:13.004885 32615 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:12.921535 3203 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.986250 3305 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:12.951666 3284 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:12.977780 3054 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.018329 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:12.993531 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:12.923508 3231 consensus_queue.cc:237] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.025442 1137 leader_election.cc:398] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:13.027622 1137 leader_election.cc:401] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:01:12.979075 3174 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.018342 3301 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:01:13.035140 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 49066.3us
I20260806 04:01:12.992818 3367 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.010677 3215 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:13.032101 32427 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:13.019721 3140 raft_consensus.cc:686] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:13.022891 3230 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.041374 3259 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:13.040903 3244 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.048271 1188 raft_consensus.cc:2354] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3.
I20260806 04:01:13.050405 3244 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:13.051864 3303 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.050287 3278 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:13.055981 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:13.059409 32469 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:13.057854 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:13.065959 32469 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:13.068259 32683 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:13.037266 3301 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:13.036883 3261 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:13.074259 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:13.074824 32477 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:13.023337 3305 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.059803 3303 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:13.087783 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 55790.4us
I20260806 04:01:13.060918 3208 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:13.041365 3031 consensus_queue.cc:1035] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 04:01:13.072825 3301 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:13.094530 32326 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39754 (request call id 339) took 1669 ms. Trace:
I20260806 04:01:13.072930 3188 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:13.099215 32682 leader_election.cc:398] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3.
I20260806 04:01:13.034950 3054 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:13.062438 3230 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.120549 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:13.123106 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:13.123839 32608 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:13.080489 3261 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.115921 3188 raft_consensus.cc:686] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:13.108194 1183 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:13.106631 3140 consensus_queue.cc:237] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.091315 3303 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:13.143471 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 81859.4us
I20260806 04:01:13.153024 32427 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:13.156461 32322 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36880 (request call id 230) took 1651 ms. Trace:
I20260806 04:01:13.158632 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:13.158862 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:13.159540 1188 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:01:13.159289 1189 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:13.162201 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:13.162756 32615 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:13.142469 3200 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.102151 3208 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.166083 32551 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:13.201305 3174 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:13.202924 32749 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:01:13.208278 32469 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 7 index: 10. (index mismatch)
I20260806 04:01:13.098307 32326 rpcz_store.cc:276] 0806 04:01:11.424746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:11.426357 (+ 1611us) service_pool.cc:227] Handling call
0806 04:01:13.094461 (+1668104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:11.631139 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:11.755088 (+123949us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:11.777255 (+ 22167us) write_op.cc:435] Acquired schema lock
0806 04:01:11.777353 (+ 98us) tablet.cc:660] Decoding operations
0806 04:01:11.778019 (+ 666us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:11.782091 (+ 4072us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:11.792952 (+ 10861us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:01:11.894486 (+101534us) tablet.cc:699] Row locks acquired
0806 04:01:11.907472 (+ 12986us) write_op.cc:260] PREPARE: finished
0806 04:01:11.934255 (+ 26783us) write_op.cc:270] Start()
0806 04:01:11.992218 (+ 57963us) write_op.cc:276] Timestamp: P: 1785988871925688 usec, L: 0
0806 04:01:11.992311 (+ 93us) op_driver.cc:331] REPLICATION: starting
0806 04:01:11.993342 (+ 1031us) log.cc:842] Serialized 544 byte log entry
0806 04:01:12.639535 (+646193us) op_driver.cc:444] REPLICATION: finished
0806 04:01:12.797933 (+158398us) write_op.cc:301] APPLY: starting
0806 04:01:12.801833 (+ 3900us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:12.818407 (+ 16574us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:12.832007 (+ 13600us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:12.965951 (+133944us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:13.026182 (+ 60231us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:01:13.029421 (+ 3239us) write_op.cc:312] APPLY: finished
0806 04:01:13.040637 (+ 11216us) log.cc:842] Serialized 23 byte log entry
0806 04:01:13.040852 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:13.040953 (+ 101us) write_op.cc:454] Released schema lock
0806 04:01:13.041080 (+ 127us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:13.073538 (+ 32458us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:13.078806 (+ 5268us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":136873,"apply.run_cpu_time_us":8526,"apply.run_wall_time_us":304267,"num_ops":1,"prepare.queue_time_us":163714,"prepare.run_cpu_time_us":59376,"prepare.run_wall_time_us":525035,"raft.queue_time_us":326582,"raft.run_cpu_time_us":70705,"raft.run_wall_time_us":202878,"replication_time_us":647032,"thread_start_us":428972,"threads_started":4,"wal-append.queue_time_us":178777}]]}
I20260806 04:01:13.149068 3353 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.148423 3197 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:13.155009 3219 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:13.161466 3285 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:13.156821 32322 rpcz_store.cc:276] 0806 04:01:11.504662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:11.504822 (+ 160us) service_pool.cc:227] Handling call
0806 04:01:13.156401 (+1651579us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:12.150230 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:12.263631 (+113401us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:12.288469 (+ 24838us) write_op.cc:435] Acquired schema lock
0806 04:01:12.292204 (+ 3735us) tablet.cc:660] Decoding operations
0806 04:01:12.372842 (+ 80638us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:12.372936 (+ 94us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:12.372984 (+ 48us) tablet.cc:683] Acquiring locks for 2 operations
0806 04:01:12.422423 (+ 49439us) tablet.cc:699] Row locks acquired
0806 04:01:12.422465 (+ 42us) write_op.cc:260] PREPARE: finished
0806 04:01:12.422687 (+ 222us) write_op.cc:270] Start()
0806 04:01:12.422853 (+ 166us) write_op.cc:276] Timestamp: P: 1785988872422619 usec, L: 0
0806 04:01:12.422900 (+ 47us) op_driver.cc:331] REPLICATION: starting
0806 04:01:12.423817 (+ 917us) log.cc:842] Serialized 925 byte log entry
0806 04:01:12.872370 (+448553us) op_driver.cc:444] REPLICATION: finished
0806 04:01:13.001692 (+129322us) write_op.cc:301] APPLY: starting
0806 04:01:13.001829 (+ 137us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:13.002048 (+ 219us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:13.025617 (+ 23569us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:13.055142 (+ 29525us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:13.075476 (+ 20334us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 04:01:13.084062 (+ 8586us) write_op.cc:312] APPLY: finished
0806 04:01:13.101139 (+ 17077us) log.cc:842] Serialized 29 byte log entry
0806 04:01:13.115495 (+ 14356us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:13.121561 (+ 6066us) write_op.cc:454] Released schema lock
0806 04:01:13.127132 (+ 5571us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":128954,"apply.run_cpu_time_us":4011,"apply.run_wall_time_us":173031,"num_ops":2,"prepare.queue_time_us":613410,"prepare.run_cpu_time_us":5217,"prepare.run_wall_time_us":278436,"replication_time_us":449331,"thread_start_us":38490,"threads_started":1}]]}
I20260806 04:01:13.161877 3284 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:13.167220 3203 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.266686 32324 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39766 (request call id 336) took 1129 ms. Trace:
I20260806 04:01:13.267371 32324 rpcz_store.cc:276] 0806 04:01:12.134321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:12.144494 (+ 10173us) service_pool.cc:227] Handling call
0806 04:01:13.263456 (+1118962us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260806 04:01:13.283893 3208 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:13.285269 32737 consensus_meta.cc:287] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.610s user 0.007s sys 0.005s
I20260806 04:01:13.285656 32737 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:13.247282 3219 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.244812 3142 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260806 04:01:13.287781 3219 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:13.246887 3285 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:13.242057 3144 raft_consensus.cc:348] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:13.245329 3197 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.223429 3379 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:13.242311 3359 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:13.290818 32469 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:01:13.246991 1188 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:13.290416 32615 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:13.202977 3200 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.243264 3257 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260806 04:01:13.256951 1183 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:13.301415 32749 raft_consensus.cc:1261] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:01:13.297739 32428 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:13.279287 3244 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.309150 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:13.309875 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:13.310719 3230 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.304669 3379 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:13.311400 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:13.315375 32477 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:13.293083 3305 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.202268 3188 consensus_queue.cc:237] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.296097 3285 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:13.289738 3367 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:13.294471 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:13.329077 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:13.335152 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:13.334723 1137 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:13.295528 3144 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:01:13.342553 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:13.343223 1136 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:13.348933 32477 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:13.352123 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:13.348407 32605 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:13.350638 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:13.349560 1189 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:13.362375 32749 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:13.321069 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:13.326283 3158 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.334743 3379 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:13.320612 3405 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:13.332959 32740 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:13.344597 3303 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.341497 32738 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:13.330421 3244 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:13.470525 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 173402us
W20260806 04:01:13.495442 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 158416us
I20260806 04:01:13.319917 3359 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.339785 3366 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.376394 3261 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:13.532776 32477 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:13.452996 3203 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:13.659224 32553 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:13.659602 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:13.665400 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:13.667320 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:13.672605 1188 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:01:13.660333 32469 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:13.452553 32602 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:13.667927 32466 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:13.677600 32743 raft_consensus.cc:1261] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:13.467208 3259 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:13.680495 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:13.680629 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 337462us
I20260806 04:01:13.683732 32470 raft_consensus.cc:2354] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:13.510885 3283 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:13.684695 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:13.688401 32603 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:13.469910 3134 consensus_queue.cc:1035] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.220s
I20260806 04:01:13.693594 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:13.434100 3158 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.750188 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:13.754127 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 60627.3us
I20260806 04:01:13.681646 3203 raft_consensus.cc:686] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:13.778911 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:13.790344 32604 raft_consensus.cc:2379] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:13.683941 3367 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:13.501754 3405 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.647603 3366 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.693562 3215 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:13.750984 32605 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:13.748327 3319 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:13.822204 3274 consensus_meta.cc:287] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.939s user 0.003s sys 0.012s
I20260806 04:01:13.503082 3144 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:13.783143 3321 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:13.661480 3416 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.701432 3283 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:13.856215 3278 consensus_meta.cc:287] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.785s user 0.010s sys 0.000s
I20260806 04:01:13.845945 3200 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.850392 3197 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:13.875156 3201 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:13.904356 3218 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:13.888851 3140 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:13.899660 3331 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.852592 3319 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:13.903311 3278 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:13.876986 3274 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.908752 3200 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:13.910902 3201 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:13.933470 32466 raft_consensus.cc:1261] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:13.902565 3134 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260806 04:01:13.958500 3359 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:13.959668 3366 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:13.960799 3319 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:13.981180 32469 raft_consensus.cc:1226] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:13.961032 3405 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.884442 3321 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:13.961683 3201 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:13.963441 3433 log.cc:924] Time spent T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260806 04:01:13.963366 3197 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:13.959688 3283 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.969080 32321 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39782 (request call id 331) took 2324 ms. Trace:
I20260806 04:01:13.470861 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:13.978418 3292 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:13.969744 3301 consensus_meta.cc:287] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.858s user 0.012s sys 0.000s
I20260806 04:01:13.919356 3140 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:13.931768 3152 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.122s user 0.014s sys 0.000s
I20260806 04:01:13.983603 3244 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:13.900807 3416 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.919661 3218 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:13.992794 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:13.926419 3331 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:13.979214 32688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.031s sys 0.061s
I20260806 04:01:13.917331 3203 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:14.018179 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.051s sys 0.093s
I20260806 04:01:14.020344 32470 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
W20260806 04:01:14.039299 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.526s user 0.051s sys 0.093s
I20260806 04:01:14.020782 32608 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:13.973497 3200 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:14.023790 32688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.031s sys 0.061s
I20260806 04:01:14.045359 32683 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:14.038460 3218 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:14.048588 3416 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.049376 32428 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:14.005780 3158 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.015311 3140 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:13.994558 32321 rpcz_store.cc:276] 0806 04:01:11.644447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:11.659013 (+ 14566us) service_pool.cc:227] Handling call
0806 04:01:13.969010 (+2309997us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:12.461553 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:12.504017 (+ 42464us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:12.504044 (+ 27us) write_op.cc:435] Acquired schema lock
0806 04:01:12.504064 (+ 20us) tablet.cc:660] Decoding operations
0806 04:01:12.504445 (+ 381us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:12.504491 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:12.504516 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:01:12.504691 (+ 175us) tablet.cc:699] Row locks acquired
0806 04:01:12.504707 (+ 16us) write_op.cc:260] PREPARE: finished
0806 04:01:12.504835 (+ 128us) write_op.cc:270] Start()
0806 04:01:12.504946 (+ 111us) write_op.cc:276] Timestamp: P: 1785988872504787 usec, L: 0
0806 04:01:12.504966 (+ 20us) op_driver.cc:331] REPLICATION: starting
0806 04:01:12.548837 (+ 43871us) log.cc:842] Serialized 603 byte log entry
0806 04:01:12.944556 (+395719us) op_driver.cc:444] REPLICATION: finished
0806 04:01:13.090109 (+145553us) write_op.cc:301] APPLY: starting
0806 04:01:13.095208 (+ 5099us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:13.108894 (+ 13686us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:13.128751 (+ 19857us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:13.161984 (+ 33233us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:13.224042 (+ 62058us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:01:13.232842 (+ 8800us) write_op.cc:312] APPLY: finished
0806 04:01:13.263929 (+ 31087us) log.cc:842] Serialized 23 byte log entry
0806 04:01:13.290032 (+ 26103us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:13.295358 (+ 5326us) write_op.cc:454] Released schema lock
0806 04:01:13.297980 (+ 2622us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:13.329690 (+ 31710us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:13.334791 (+ 5101us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":131565,"apply.run_cpu_time_us":3085,"apply.run_wall_time_us":268350,"num_ops":1,"prepare.queue_time_us":521934,"prepare.run_cpu_time_us":3071,"prepare.run_wall_time_us":92425,"replication_time_us":439388,"thread_start_us":72053,"threads_started":1}]]}
I20260806 04:01:14.048370 3200 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:13.995512 3197 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.055809 3405 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:14.074803 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:14.043604 3305 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.077713 3305 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:14.043888 3283 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.015748 3292 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:14.062219 3152 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.098872 3405 raft_consensus.cc:686] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:14.015548 3301 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.105424 32749 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 04:01:14.116642 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 377763us
W20260806 04:01:14.116930 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 294281us
I20260806 04:01:14.060849 3435 consensus_queue.cc:1046] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260806 04:01:14.122423 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.123167 32469 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:14.124454 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.106640 3049 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:14.128739 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.131736 32603 raft_consensus.cc:2379] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:14.141418 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:14.142123 1189 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:14.145650 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.146524 32603 raft_consensus.cc:2379] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:14.135542 3353 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.114063 3172 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.118731 3429 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.120980 3367 raft_consensus.cc:686] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:14.148954 32553 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:14.150197 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.155634 32610 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:14.100021 3283 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.114401 3274 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.152448 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.101881 3152 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.154810 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:14.148190 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:14.164506 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.164947 1188 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:14.165899 1192 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
W20260806 04:01:14.159693 32473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.017s sys 0.008s
I20260806 04:01:14.136343 3049 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:14.166720 32470 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
W20260806 04:01:14.167246 32473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s user 0.018s sys 0.009s
I20260806 04:01:14.167940 1137 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:14.113806 3278 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260806 04:01:14.173519 3049 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.180426 3292 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:14.182744 3447 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.183171 3449 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:14.183917 3449 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:14.184468 3449 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.183490 3447 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.189875 3283 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.191136 3447 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.162782 3172 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.156507 3405 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.444656 32609 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:14.446087 3172 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.201531 3301 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.197044 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:14.192783 3447 consensus_queue.cc:1035] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.255s
I20260806 04:01:14.446372 3305 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.446820 3140 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.156415 3353 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.451607 3140 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.306871 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:14.435863 3176 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.163007 3429 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.442262 3144 ts_tablet_manager.cc:1428] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 1.556s user 0.024s sys 0.017s
I20260806 04:01:14.306793 3049 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.198275 3197 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:14.199172 1183 consensus_meta.cc:287] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.879s user 0.009s sys 0.003s
I20260806 04:01:14.253144 1137 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
W20260806 04:01:14.286271 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 117994us
I20260806 04:01:14.202155 3434 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:14.175793 3331 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:14.168399 32427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.005s sys 0.013s
I20260806 04:01:14.171576 32605 raft_consensus.cc:2379] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:14.266270 3367 consensus_queue.cc:237] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.174387 3321 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:14.207604 3430 log.cc:924] Time spent T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.291s user 0.000s sys 0.000s
I20260806 04:01:14.436679 32324 catalog_manager.cc:5495] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 2, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 2 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:14.375299 32738 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:14.307425 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 1936) took 1011 ms. Trace:
I20260806 04:01:14.281572 3152 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.347608 3280 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:14.353210 3201 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.281867 3448 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.451998 3353 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.493793 3448 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:14.455948 3144 ts_tablet_manager.cc:1460] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Tablet startup took 1569ms
W20260806 04:01:14.349695 3285 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.992s user 0.015s sys 0.003s
I20260806 04:01:14.457858 32614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:41296 (request call id 1083) took 1638 ms. Trace:
I20260806 04:01:14.453969 3176 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.461781 1136 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:14.463234 32427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.065s user 0.009s sys 0.017s
I20260806 04:01:14.465721 3434 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:14.461946 3054 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.490365 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1512) took 1319 ms. Trace:
I20260806 04:01:14.361526 3437 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.478209 32608 rpcz_store.cc:276] 0806 04:01:13.296335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:13.311153 (+ 14818us) service_pool.cc:227] Handling call
0806 04:01:14.307237 (+996084us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54003359d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:14.307359 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1591401728}
I20260806 04:01:14.460386 1183 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 7 index: 10. (index mismatch)
I20260806 04:01:14.467334 32740 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:14.537664 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.539940 32603 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:14.490954 3429 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.369463 3455 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:14.504680 32614 rpcz_store.cc:276] 0806 04:01:12.819769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:12.819886 (+ 117us) service_pool.cc:227] Handling call
0806 04:01:14.457133 (+1637247us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54003359d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d25cf2a 00007fd172199c6e 00007fd16b890807 00007fd16b890792 00007fd16b8906c1 00007fd16b89063c 00007fd16b88f8f2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:14.457735 (+ 602us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2698610560}
W20260806 04:01:14.506080 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 69779.7us
I20260806 04:01:14.459429 3049 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:14.495263 3201 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:14.513190 3215 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.042s sys 0.036s
I20260806 04:01:14.376713 3231 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:14.519755 3303 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.009s sys 0.025s
W20260806 04:01:14.606748 3303 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.928s user 0.009s sys 0.025s
I20260806 04:01:14.492376 3280 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.526085 3218 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:14.504634 3379 consensus_meta.cc:287] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.036s user 0.007s sys 0.014s
I20260806 04:01:14.531867 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.527460 32473 rpcz_store.cc:276] 0806 04:01:13.157193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:13.157350 (+ 157us) service_pool.cc:227] Handling call
0806 04:01:13.306026 (+148676us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:13.306875 (+ 849us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:13.306904 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:13.307283 (+ 379us) log.cc:842] Serialized 26 byte log entry
0806 04:01:14.173336 (+866053us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:14.173384 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:14.173469 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:14.476391 (+302922us) raft_consensus.cc:1671] finished
0806 04:01:14.476409 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:14.476528 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":521,"raft.queue_time_us":89798,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":73371,"thread_start_us":825694,"threads_started":1,"wal-append.queue_time_us":976207}
W20260806 04:01:14.533802 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 280717us
I20260806 04:01:14.633942 1137 leader_election.cc:398] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
W20260806 04:01:14.533967 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 421866us
W20260806 04:01:14.634398 32741 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.031s sys 0.037s
W20260806 04:01:14.634719 32741 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s user 0.031s sys 0.037s
W20260806 04:01:14.533561 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 415343us
I20260806 04:01:14.637529 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.638559 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.639189 32469 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 1.
I20260806 04:01:14.640228 1137 leader_election.cc:401] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:14.536881 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.537129 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.647902 32609 raft_consensus.cc:2379] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:14.497210 3144 ts_tablet_manager.cc:1462] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Trace:
0806 04:01:07.465228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:07.469325 (+ 4097us) service_pool.cc:227] Handling call
0806 04:01:07.997710 (+528385us) ts_tablet_manager.cc:928] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
0806 04:01:10.033281 (+2035571us) tablet_copy_client.cc:361] Tablet copy session begun
0806 04:01:10.758843 (+725562us) tablet_metadata.cc:641] Metadata flushed
0806 04:01:10.759780 (+ 937us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 04:01:11.078496 (+318716us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 04:01:11.088416 (+ 9920us) tablet_replica.cc:205] Creating consensus instance
0806 04:01:11.088900 (+ 484us) ts_tablet_manager.cc:983] Replica registered.
0806 04:01:11.089051 (+ 151us) inbound_call.cc:173] Queueing success response
0806 04:01:11.426950 (+337899us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0806 04:01:12.886418 (+1459468us) ts_tablet_manager.cc:1429] Starting tablet replica
0806 04:01:13.047782 (+161364us) tablet_replica.cc:259] Starting instrumentation
0806 04:01:13.048820 (+ 1038us) tablet_replica.cc:278] Starting consensus
0806 04:01:14.371617 (+1322797us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x7b48000d05d8. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1720fb7f1 00007fd1720f28e7 00007fd1720f277a 00007fd1721e695f 00007fd1721e6627 00007fd1721fab2c 00007fd1721faad8 00007fd1721faa58 00007fd1721faa13 00007fd1721f9cbc 00007fd17117bbe6
Metrics: {"dns_us":1032,"mutex_wait_us":70657,"raft.queue_time_us":46560,"raft.run_cpu_time_us":3284,"raft.run_wall_time_us":445262,"spinlock_wait_cycles":11100544,"tablet-copy.queue_time_us":511090,"tablet-download-pool.queue_time_us":294,"tablet-download-pool.run_cpu_time_us":1021,"tablet-download-pool.run_wall_time_us":109694,"thread_start_us":2280961,"threads_started":3}
W20260806 04:01:14.552511 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 90798.6us
I20260806 04:01:14.552460 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.529170 3434 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.605620 3414 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:14.555707 3215 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.639s user 0.043s sys 0.036s
I20260806 04:01:14.606117 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:14.506901 3285 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.533139 3353 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:14.538048 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.663272 32602 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:14.663359 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:14.663978 1187 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.665130 1136 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:14.669332 3197 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:14.669677 3197 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:14.669972 3197 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:14.670802 1187 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:14.642829 3259 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:14.671561 3259 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.672250 3450 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:14.673494 3259 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.675107 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.677973 3176 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.678671 3319 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.679105 3319 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.644747 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:14.680325 1185 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.681331 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.682015 32477 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:14.682910 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.683480 32477 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:14.650293 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.555302 3455 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.684516 32470 raft_consensus.cc:2379] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:14.685987 3455 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.650781 32466 raft_consensus.cc:2396] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260806 04:01:14.689148 3464 raft_consensus.cc:1050] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:01:14.538301 3413 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:14.628834 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 694) took 1454 ms. Trace:
I20260806 04:01:14.628090 3049 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.655603 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:14.656070 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:14.656379 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 507482us
I20260806 04:01:14.726253 32475 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.655117 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:14.558177 3331 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.727420 3331 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.728729 32743 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:14.657017 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.729409 32603 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:14.658842 32471 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:01:14.606071 3231 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.679358 3176 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.629004 3158 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:14.692703 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:14.693346 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:14.652544 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.651244 3152 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:14.655798 3215 raft_consensus.cc:686] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:14.630632 3218 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.734968 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1617) took 1219 ms. Trace:
I20260806 04:01:14.696126 32683 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:14.606030 3452 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:14.706794 1185 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:14.704309 1137 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:14.729658 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.724033 32478 raft_consensus.cc:2354] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:14.537308 3437 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.726730 32553 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:14.767299 32609 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:14.767972 32746 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:14.693835 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.745680 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:14.769435 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 65224.8us
I20260806 04:01:14.642568 3428 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.745329 32474 raft_consensus.cc:1226] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 04:01:14.767741 32748 raft_consensus.cc:2354] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:14.806720 32477 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:14.630818 3379 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:14.744479 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 110676us
I20260806 04:01:14.769153 32614 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
W20260806 04:01:14.801295 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 74645.6us
I20260806 04:01:14.690476 3321 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.712255 3413 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:14.818806 32475 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
W20260806 04:01:14.765307 3449 consensus_meta.cc:287] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.580s user 0.012s sys 0.000s
W20260806 04:01:14.819083 32467 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.020s sys 0.033s
I20260806 04:01:14.759196 3448 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.770561 32741 rpcz_store.cc:276] 0806 04:01:13.515740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:13.515854 (+ 114us) service_pool.cc:227] Handling call
0806 04:01:13.677354 (+161500us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:13.917157 (+239803us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:13.917184 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:13.923280 (+ 6096us) log.cc:842] Serialized 26 byte log entry
0806 04:01:14.635108 (+711828us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:14.635146 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:14.635210 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:14.734775 (+ 99565us) raft_consensus.cc:1671] finished
0806 04:01:14.734793 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:14.734916 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":144777,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":9543,"spinlock_wait_cycles":2944,"thread_start_us":915845,"threads_started":2,"wal-append.queue_time_us":807574}
I20260806 04:01:14.713219 3438 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:14.751652 3331 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.809268 3447 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:14.745704 3462 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.750679 3455 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.745467 3283 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20260806 04:01:14.814218 32428 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:14.828719 3413 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:14.831189 1137 leader_election.cc:398] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:14.744274 3172 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:14.775489 3457 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.751336 3432 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:14.837031 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.879055 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:14.884714 32475 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:14.839417 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.757880 3319 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.743309 3274 raft_consensus.cc:3032] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:14.886039 3274 raft_consensus.cc:729] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:14.885783 3215 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.886945 3274 consensus_queue.cc:260] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.887979 3274 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.844648 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.845795 32743 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:01:14.890216 32743 raft_consensus.cc:595] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:14.657657 3414 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.847013 32604 raft_consensus.cc:2354] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:14.838476 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:14.827803 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 130426us
I20260806 04:01:14.893035 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:14.894101 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:14.895346 3414 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:14.851033 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.896667 32614 raft_consensus.cc:2396] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260806 04:01:14.779858 3452 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:14.775708 3280 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.897832 1188 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:14.897768 3188 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:14.889911 32477 raft_consensus.cc:2379] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:14.897523 3452 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:14.899502 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:14.849884 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 153864us
I20260806 04:01:14.704417 3353 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.904560 3353 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:14.827488 32467 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.020s sys 0.033s
I20260806 04:01:14.906137 3231 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.865145 3462 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:14.827443 3203 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.004s sys 0.057s
W20260806 04:01:14.909554 3203 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s user 0.004s sys 0.057s
I20260806 04:01:14.853322 3285 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.859282 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.910148 3462 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.853832 3447 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:14.910629 32478 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:14.830566 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:14.866286 32610 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
W20260806 04:01:14.868016 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 53878.9us
I20260806 04:01:14.869406 1137 leader_election.cc:401] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260806 04:01:14.914036 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 83002.4us
I20260806 04:01:14.867072 3455 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.870797 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.918726 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:14.774708 3158 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.776211 3049 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.918289 3455 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.712136 1183 rpcz_store.cc:276] 0806 04:01:13.166824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:13.174495 (+ 7671us) service_pool.cc:227] Handling call
0806 04:01:13.247011 (+ 72516us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:14.546730 (+1299719us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:14.621541 (+ 74811us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:14.774428 3176 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.853977 3134 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.864065 3469 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:14.804323 3152 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.879004 3439 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.840837 3449 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.811584 3428 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.880559 3218 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:14.882032 3379 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:14.885069 32602 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.884418 3432 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.889855 3464 consensus_queue.cc:237] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.910985 3447 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.896351 3331 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:14.921183 3172 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.923795 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:14.925511 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.915829 1137 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:14.925820 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:14.924535 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:14.927693 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:14.924652 32683 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:14.935781 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:14.941857 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:14.942893 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:14.943907 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:14.945045 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.945791 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:14.871898 3230 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:14.828266 3321 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.850165 3140 consensus_queue.cc:1035] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
W20260806 04:01:14.943446 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 277542us
I20260806 04:01:14.949553 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:14.950562 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.934417 3285 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:14.936403 3414 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:14.854547 3438 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:14.939489 3319 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260806 04:01:14.981810 32604 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 04:01:15.125025 32471 raft_consensus.cc:2379] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:15.052702 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.940189 3452 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.127281 1136 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:15.128268 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 212508us
I20260806 04:01:15.126730 32610 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:15.126029 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 183206us
I20260806 04:01:14.944160 3280 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:14.941425 3188 raft_consensus.cc:686] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:15.135393 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 210791us
I20260806 04:01:15.149405 3331 raft_consensus.cc:686] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:15.154074 3280 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.152571 3414 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:14.946961 3134 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.161640 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:15.161917 3353 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:15.161954 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:15.162876 32609 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:15.159371 32466 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:15.164565 3188 consensus_queue.cc:237] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.168412 3477 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:15.169073 3477 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:15.169238 3134 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.128829 3437 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.153419 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1728) took 2008 ms. Trace:
I20260806 04:01:15.149178 3261 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:15.171902 3261 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:15.172209 3477 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:15.154477 3331 consensus_queue.cc:237] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:15.135072 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 193314us
I20260806 04:01:15.156001 3455 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:15.173367 3172 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:15.174363 3452 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:15.105090 32551 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 1.686s (SENT) [suppressed 4 similar messages]
I20260806 04:01:15.188318 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:14.942835 3469 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:14.952926 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:15.180927 32739 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:14.981658 3200 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:15.126827 3230 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:15.197820 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:15.184653 32427 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:15.185444 32610 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:15.208148 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:15.148720 32477 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:01:15.187863 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:15.185899 32609 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:15.192524 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:15.209620 32746 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:15.195272 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:15.210919 32682 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:15.215718 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:15.159765 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 236035us
I20260806 04:01:15.145016 3440 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:15.226670 32427 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:15.135308 3203 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:15.202502 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:15.172379 3261 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:15.172582 3353 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:15.156478 3344 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:15.217090 32553 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:14.953194 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:15.223981 1137 leader_election.cc:398] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:15.281533 1137 leader_election.cc:401] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:15.176030 3342 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:15.177237 3321 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:15.271406 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 72617.8us
I20260806 04:01:15.275170 32477 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
W20260806 04:01:15.275360 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 58297us
I20260806 04:01:15.155544 3414 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:14.957171 3439 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.200116 3469 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:14.950374 3463 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.321s
I20260806 04:01:15.146667 3285 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.264608 3054 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:15.354612 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:15.399528 32477 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:15.400484 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:15.223839 3230 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:15.264155 3468 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:15.271343 3303 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:15.422850 32748 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:15.271126 3467 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:15.268935 1184 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:15.270291 3203 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:15.303826 3158 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:15.272549 3353 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:15.400825 3176 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.292454 3134 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:15.399006 3178 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20260806 04:01:15.416319 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:15.194162 3455 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:15.295751 3483 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:15.294606 3172 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:15.390769 3344 raft_consensus.cc:686] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:15.403820 3054 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:15.399581 3152 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.223790 3200 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.466149 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:15.394690 3049 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:15.470619 3467 raft_consensus.cc:686] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:15.469280 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 364206us
I20260806 04:01:15.478641 3158 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:15.403534 3428 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.442263 3462 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:15.475658 32477 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:15.442764 3367 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 6
I20260806 04:01:15.423619 3303 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:15.426451 3140 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:15.304594 3342 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:15.469928 3274 consensus_meta.cc:287] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.581s user 0.010s sys 0.001s
I20260806 04:01:15.470150 3203 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:15.498304 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 256359us
I20260806 04:01:15.474818 3379 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:15.404040 3283 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:15.531879 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:15.487545 3172 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:15.498050 3054 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:15.470129 32475 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.009s sys 0.052s
W20260806 04:01:15.534756 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 307160us
I20260806 04:01:15.170526 32612 rpcz_store.cc:276] 0806 04:01:13.144998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:13.148614 (+ 3616us) service_pool.cc:227] Handling call
0806 04:01:13.164513 (+ 15899us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:13.174318 (+ 9805us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:13.200226 (+ 25908us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:14.385102 (+1184876us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b2c000002e8. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:14.385717 (+ 615us) log.cc:842] Serialized 26 byte log entry
0806 04:01:14.824314 (+438597us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:15.151401 (+327087us) log.cc:842] Serialized 14 byte log entry
0806 04:01:15.152955 (+ 1554us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:15.153047 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:15.153089 (+ 42us) raft_consensus.cc:1671] finished
0806 04:01:15.153188 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:15.153330 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7046,"raft.queue_time_us":10891,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":48222,"spinlock_wait_cycles":8074112,"thread_start_us":700477,"threads_started":2,"wal-append.queue_time_us":481143}
I20260806 04:01:15.493884 3231 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:15.564349 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 340484us
I20260806 04:01:15.563616 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-d24bcc94fb734c95849f4cb89c15ca8e received from {username='slave'} at 127.0.0.1:42618
I20260806 04:01:15.523929 3152 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:15.525552 3280 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.525624 3449 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:15.519102 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:15.564967 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:15.567931 32682 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:15.486622 3134 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.584048 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:15.579371 32683 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:15.531517 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:15.570739 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:15.585762 32427 leader_election.cc:398] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:15.643146 32427 leader_election.cc:401] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260806 04:01:15.586182 32475 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.630s user 0.010s sys 0.052s
I20260806 04:01:15.588848 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-d24bcc94fb734c95849f4cb89c15ca8e on tablet d24bcc94fb734c95849f4cb89c15ca8e with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:01:15.583895 3283 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:15.590121 1137 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:15.644789 3283 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:15.645072 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 56137.1us
I20260806 04:01:15.592180 3464 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:15.648722 3134 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.648551 3464 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.563278 3303 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:15.569413 3457 consensus_meta.cc:287] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.690s user 0.002s sys 0.016s
W20260806 04:01:15.585210 3479 log.cc:924] Time spent T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.461s user 0.001s sys 0.001s
I20260806 04:01:15.609366 3455 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:15.600564 3152 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:15.629156 3491 log.cc:924] Time spent T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.358s user 0.000s sys 0.001s
I20260806 04:01:15.633157 3467 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.635501 1187 raft_consensus.cc:1226] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
W20260806 04:01:15.570576 3470 log.cc:924] Time spent T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.467s user 0.001s sys 0.001s
W20260806 04:01:15.585480 3480 log.cc:924] Time spent T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.471s user 0.001s sys 0.000s
I20260806 04:01:15.615626 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:15.544250 3158 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:01:16.098384 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 519088us
W20260806 04:01:16.041653 3469 consensus_meta.cc:287] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.628s user 0.007s sys 0.003s
I20260806 04:01:16.144307 32683 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:16.031446 3502 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:16.145361 3502 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.079581 32749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 333) took 1377 ms. Trace:
I20260806 04:01:15.588320 3172 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:16.080847 3178 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20260806 04:01:15.596727 3432 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.704737 32746 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:16.111099 1186 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260806 04:01:16.098503 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 530604us
I20260806 04:01:16.109709 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1733) took 1455 ms. Trace:
I20260806 04:01:16.114599 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:15.650484 3134 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.512s
I20260806 04:01:16.163867 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:15.570257 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.007s sys 0.014s
W20260806 04:01:16.164752 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.117s user 0.007s sys 0.019s
W20260806 04:01:16.165014 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.11695e+06us
W20260806 04:01:16.148329 32688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.003s sys 0.042s
W20260806 04:01:16.166051 32688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.004s sys 0.042s
I20260806 04:01:15.812050 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:15.998301 3498 log.cc:924] Time spent T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.413s user 0.001s sys 0.000s
W20260806 04:01:15.663980 3197 consensus_meta.cc:287] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.994s user 0.013s sys 0.000s
I20260806 04:01:16.173084 32477 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:16.148494 32749 rpcz_store.cc:276] 0806 04:01:14.701748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.703522 (+ 1774us) service_pool.cc:227] Handling call
0806 04:01:14.703822 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:14.952625 (+248803us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:14.952654 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:14.952692 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:14.952725 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:16.079262 (+1126537us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b540011dad4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:16.079313 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.079499 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":571718016}
I20260806 04:01:16.222822 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:16.223436 32749 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:15.597498 3438 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:16.161691 32608 rpcz_store.cc:276] 0806 04:01:14.654183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.654316 (+ 133us) service_pool.cc:227] Handling call
0806 04:01:14.654589 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:14.999434 (+344845us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:14.999468 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:14.999509 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:14.999540 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:16.109466 (+1109926us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b54002d0354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:16.109505 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.109642 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":792487296}
W20260806 04:01:16.172852 32602 consensus_meta.cc:287] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.125s user 0.005s sys 0.007s
I20260806 04:01:16.099102 3049 raft_consensus.cc:2781] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:15.812968 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:16.227788 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:15.812567 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:15.816030 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:16.230014 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1631) took 1032 ms. Trace:
W20260806 04:01:15.653115 3494 log.cc:924] Time spent T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20260806 04:01:16.040534 3200 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:15.663221 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 77605.7us
I20260806 04:01:16.112159 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 328) took 1420 ms. Trace:
I20260806 04:01:16.235416 1192 raft_consensus.cc:2379] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 2: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:15.901423 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:16.217366 3415 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:16.208891 3230 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.785s user 0.009s sys 0.002s
I20260806 04:01:15.595818 3231 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:16.029500 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:16.098672 3176 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:15.653877 3487 log.cc:924] Time spent T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.265s user 0.000s sys 0.001s
W20260806 04:01:16.219188 32470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.018s sys 0.022s
W20260806 04:01:15.666214 3473 log.cc:924] Time spent T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.783s user 0.008s sys 0.006s
I20260806 04:01:16.159693 32609 raft_consensus.cc:2379] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:16.042532 3434 consensus_meta.cc:287] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.348s user 0.006s sys 0.006s
I20260806 04:01:15.695750 3218 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:16.104557 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1525) took 1266 ms. Trace:
I20260806 04:01:16.227769 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:16.230535 32739 rpcz_store.cc:276] 0806 04:01:15.197699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.221983 (+ 24284us) service_pool.cc:227] Handling call
0806 04:01:15.520725 (+298742us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b5800004298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:15.815881 (+295156us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b5800004298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4b7c 00007fd16a2f473b 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:16.229951 (+414070us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101257216}
I20260806 04:01:15.667954 3437 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:16.099314 3319 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:15.700271 3342 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:16.232483 1186 raft_consensus.cc:2379] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:15.656818 3439 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:15.656352 3414 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:15.651536 3464 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.273887 3230 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:15.651130 3449 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:16.235848 32737 rpcz_store.cc:276] 0806 04:01:14.691660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.701294 (+ 9634us) service_pool.cc:227] Handling call
0806 04:01:14.732897 (+ 31603us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:15.169353 (+436456us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:15.169410 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:15.169461 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:15.169514 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:16.111920 (+942406us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54001f2054. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:16.111960 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.112075 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1002996608}
I20260806 04:01:16.409482 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:16.150647 3457 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:15.588514 3379 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:15.567221 3471 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:15.559428 3462 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.425622 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:16.426724 3342 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:16.452306 3437 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:16.452116 3342 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.452838 3437 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.212610 3455 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:15.670048 3285 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.259400 3513 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:16.454592 3342 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.455046 3455 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.252858 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:16.260632 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:16.452171 3462 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.456226 32748 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:16.455502 3513 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:16.457300 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:16.458050 32748 raft_consensus.cc:2379] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:16.227645 32602 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
W20260806 04:01:16.459273 32602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1741) took 1586 ms (client timeout 1615 ms). Trace:
W20260806 04:01:16.459692 32602 rpcz_store.cc:269] 0806 04:01:14.873031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.874277 (+ 1246us) service_pool.cc:227] Handling call
0806 04:01:16.459179 (+1584902us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:16.225282 3452 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:16.216310 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 629) took 1287 ms. Trace:
I20260806 04:01:15.569661 3140 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.232326 3049 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:16.269160 32470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.018s sys 0.023s
I20260806 04:01:16.344157 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1721) took 1173 ms. Trace:
I20260806 04:01:16.460947 3049 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:16.460750 1187 rpcz_store.cc:276] 0806 04:01:14.928874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.929004 (+ 130us) service_pool.cc:227] Handling call
0806 04:01:14.929368 (+ 364us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:15.635255 (+705887us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b54004a04a0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d1e4730 00007fd16d256300 00007fd16d257371 00007fd16d253edf 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc
0806 04:01:16.170205 (+534950us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:16.216032 (+ 45827us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54005d1454. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:16.216242 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":506162816}
W20260806 04:01:16.268272 3511 log.cc:924] Time spent T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260806 04:01:16.461374 32478 rpcz_store.cc:276] 0806 04:01:15.170582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.170683 (+ 101us) service_pool.cc:227] Handling call
0806 04:01:15.171103 (+ 420us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:15.171329 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:15.171353 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:15.171812 (+ 459us) log.cc:842] Serialized 26 byte log entry
0806 04:01:15.403301 (+231489us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:15.403350 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:15.403424 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:16.343891 (+940467us) raft_consensus.cc:1671] finished
0806 04:01:16.343914 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.344067 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":205922,"threads_started":1,"wal-append.queue_time_us":347534}
I20260806 04:01:16.462579 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:16.312834 3278 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:16.171597 3469 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:16.379102 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:16.219327 3197 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.252530 3200 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:16.427340 32474 raft_consensus.cc:2379] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:16.426262 1186 raft_consensus.cc:2379] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:16.528520 32739 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:16.530869 1189 raft_consensus.cc:2354] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4.
I20260806 04:01:16.424568 3449 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:16.268301 3415 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.426117 3523 log.cc:924] Time spent T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20260806 04:01:16.456280 32749 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:16.252601 3440 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:15.603428 3344 consensus_queue.cc:237] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:16.332577 32466 rpcz_store.cc:276] 0806 04:01:14.838048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.839103 (+ 1055us) service_pool.cc:227] Handling call
0806 04:01:16.104321 (+1265218us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b540009cc54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:16.104432 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":629374208}
W20260806 04:01:16.483206 32559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.841s user 0.012s sys 0.023s
W20260806 04:01:16.409548 3478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s user 0.000s sys 0.029s
I20260806 04:01:16.362979 3176 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:16.403414 3319 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:16.342231 3434 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.381168 3218 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.380780 3518 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.337s
W20260806 04:01:16.275630 3331 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.002s sys 0.019s
W20260806 04:01:16.678755 32551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.079s user 0.005s sys 0.037s
I20260806 04:01:16.610096 3342 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:16.813899 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1522) took 1976 ms. Trace:
I20260806 04:01:16.813741 3342 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:16.814404 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:16.806448 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2063) took 1890 ms. Trace:
I20260806 04:01:16.538156 3278 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.808012 3176 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.759958 3261 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.464s user 0.006s sys 0.006s
W20260806 04:01:16.763187 3172 consensus_meta.cc:287] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.591s user 0.007s sys 0.000s
I20260806 04:01:16.817389 32743 rpcz_store.cc:276] 0806 04:01:14.916460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.924908 (+ 8448us) service_pool.cc:227] Handling call
0806 04:01:14.950138 (+ 25230us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:15.404311 (+454173us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:15.404343 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:15.440581 (+ 36238us) log.cc:842] Serialized 26 byte log entry
0806 04:01:15.898196 (+457615us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:15.898234 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:16.108580 (+210346us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b540019b4d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:16.108629 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:16.806164 (+697535us) raft_consensus.cc:1671] finished
0806 04:01:16.806185 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.806392 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25908,"raft.queue_time_us":78829,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":37506,"spinlock_wait_cycles":1043657600,"thread_start_us":403437,"threads_started":1,"wal-append.queue_time_us":789589}
I20260806 04:01:16.819344 1187 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:16.806150 3140 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.823052 3140 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:16.823609 3140 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.535670 3054 consensus_meta.cc:287] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.940s user 0.002s sys 0.006s
I20260806 04:01:16.825121 3054 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:16.826231 32469 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:01:16.805385 3049 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.830179 3218 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:16.534128 3469 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:16.454681 3437 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.831450 3437 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:16.832002 3437 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.833500 3532 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.140s user 0.008s sys 0.000s
I20260806 04:01:16.833981 3437 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.440006 3292 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 04:01:16.755223 3283 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 1.110s user 0.008s sys 0.002s
W20260806 04:01:16.758736 3203 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 1.171s user 0.008s sys 0.000s
I20260806 04:01:16.268721 3504 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:16.694020 32738 raft_consensus.cc:2379] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:16.812997 3215 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.891s user 0.005s sys 0.042s
W20260806 04:01:16.581269 3178 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260806 04:01:16.818764 3172 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.830735 3478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.421s user 0.004s sys 0.040s
I20260806 04:01:16.554634 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1620) took 1860 ms. Trace:
I20260806 04:01:16.861891 32745 rpcz_store.cc:276] 0806 04:01:14.694476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.696652 (+ 2176us) service_pool.cc:227] Handling call
0806 04:01:14.696947 (+ 295us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:14.818742 (+121795us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:14.818776 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:14.868410 (+ 49634us) spinlock_profiling.cc:243] Waited 8.3 ms on lock 0x7b2c00108868. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:14.874754 (+ 6344us) log.cc:842] Serialized 26 byte log entry
0806 04:01:15.159848 (+285094us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:15.159888 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:15.159958 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:16.492864 (+1332906us) raft_consensus.cc:1671] finished
0806 04:01:16.492888 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.554544 (+ 61656us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6943,"raft.queue_time_us":134555,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":8659,"spinlock_wait_cycles":19078912,"thread_start_us":243919,"threads_started":1,"wal-append.queue_time_us":267524}
I20260806 04:01:16.816479 3449 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:16.810772 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1515) took 2899 ms. Trace:
W20260806 04:01:16.812615 32551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.213s user 0.007s sys 0.038s
W20260806 04:01:16.870728 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 1.27095e+06us
I20260806 04:01:16.834613 3440 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:16.838945 3230 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:16.811930 3513 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:16.533932 3413 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.599s user 0.010s sys 0.001s
I20260806 04:01:16.816989 32603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 464) took 1280 ms. Trace:
W20260806 04:01:16.717388 3522 log.cc:924] Time spent T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.254s user 0.000s sys 0.000s
I20260806 04:01:16.460762 3452 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.746104 3353 consensus_meta.cc:287] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.263s user 0.005s sys 0.005s
W20260806 04:01:16.741966 3367 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.172s user 0.010s sys 0.000s
W20260806 04:01:16.743438 3447 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.690s user 0.005s sys 0.006s
W20260806 04:01:16.747992 3477 consensus_meta.cc:287] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.575s user 0.008s sys 0.004s
W20260806 04:01:16.812229 3331 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.180s user 0.005s sys 0.019s
W20260806 04:01:16.810920 32559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.169s user 0.020s sys 0.026s
I20260806 04:01:16.855866 32466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8417acc05db341aca15704cd05736651"
dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:60112
I20260806 04:01:16.838235 3176 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.857889 32615 raft_consensus.cc:2354] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:16.858605 1187 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:16.830495 3049 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:16.859499 32429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.833s user 0.014s sys 0.032s
I20260806 04:01:16.849006 3283 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.827903 3054 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:16.744709 3455 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.719254 3379 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.759761 3231 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:16.710384 3152 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.814291 32473 rpcz_store.cc:276] 0806 04:01:14.837219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.841490 (+ 4271us) service_pool.cc:227] Handling call
0806 04:01:14.841845 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:14.842264 (+ 419us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:14.842291 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:14.856179 (+ 13888us) log.cc:842] Serialized 26 byte log entry
0806 04:01:15.134516 (+278337us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:15.134559 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:15.160109 (+ 25550us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:16.332145 (+1172036us) raft_consensus.cc:1671] finished
0806 04:01:16.332171 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.813836 (+481665us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":95,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":9913,"thread_start_us":214072,"threads_started":1,"wal-append.queue_time_us":282430}
I20260806 04:01:16.804872 3158 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:16.506196 3502 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:16.869489 32467 rpcz_store.cc:276] 0806 04:01:13.911684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:13.911809 (+ 125us) service_pool.cc:227] Handling call
0806 04:01:13.912176 (+ 367us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:13.912492 (+ 316us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:13.912516 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:13.965373 (+ 52857us) spinlock_profiling.cc:243] Waited 6.84 ms on lock 0x7b2c00007ea8. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:13.965776 (+ 403us) log.cc:842] Serialized 26 byte log entry
0806 04:01:14.309581 (+343805us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b1800372220. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb411 00007fd16d2285cf
0806 04:01:14.905973 (+596392us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:01:14.906015 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:14.906081 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:15.665729 (+759648us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b24000389a8. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a184cf4 00007fd16a182dfb 00007fd16a211559 00007fd16a2111e5 00007fd16a21082d 00007fd16cebf46e 00007fd16ced6c0c 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551
0806 04:01:16.810519 (+1144790us) raft_consensus.cc:1671] finished
0806 04:01:16.810542 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.810690 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13457,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":13423,"spinlock_wait_cycles":129758976,"thread_start_us":596197,"threads_started":1,"wal-append.queue_time_us":917073}
I20260806 04:01:16.497985 3519 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:16.530599 3200 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.859876 3303 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.650s user 0.008s sys 0.000s
I20260806 04:01:16.818848 3278 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:16.857292 3215 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.935s user 0.005s sys 0.042s
I20260806 04:01:16.876415 32603 rpcz_store.cc:276] 0806 04:01:15.537027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.537632 (+ 605us) service_pool.cc:227] Handling call
0806 04:01:15.537918 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:16.462124 (+924206us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:16.462154 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:16.462191 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:16.462218 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:16.816766 (+354548us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b540035c854. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:16.816810 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:16.816928 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2125043968}
W20260806 04:01:16.876950 32683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.008s sys 0.010s
I20260806 04:01:16.872426 3504 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.881139 1139 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.513s user 0.019s sys 0.028s
I20260806 04:01:16.816004 3342 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:16.484040 3471 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.886080 32682 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.004s sys 0.019s
I20260806 04:01:16.858759 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:16.885926 32551 connection.cc:232] RPC call timeout handler was delayed by -1.0306666030082852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20260806 04:01:16.992471 1139 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.632s user 0.020s sys 0.029s
I20260806 04:01:16.992586 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:16.992901 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:16.993238 32609 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:16.817764 3261 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.898375 32466 raft_consensus.cc:595] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:16.860909 32475 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.155s user 0.023s sys 0.020s
I20260806 04:01:16.857142 3203 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:16.894990 3178 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:01:16.955129 3534 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:16.900265 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 703) took 1002 ms. Trace:
W20260806 04:01:16.983323 32429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.957s user 0.015s sys 0.037s
I20260806 04:01:17.008360 1187 rpcz_store.cc:276] 0806 04:01:15.898064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:16.814053 (+915989us) service_pool.cc:227] Handling call
0806 04:01:16.900178 (+ 86125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:17.009424 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:17.009120 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:17.010667 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:16.876981 1137 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:16.967375 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:16.895601 3367 raft_consensus.cc:2726] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:16.985123 3379 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:16.891290 1183 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:16.994263 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:16.996069 32475 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.291s user 0.023s sys 0.020s
I20260806 04:01:17.015307 32610 raft_consensus.cc:2379] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:17.015153 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:16.997648 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:16.918279 3440 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:16.998641 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.182s user 0.016s sys 0.045s
I20260806 04:01:16.933547 3452 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:16.897648 3447 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.027901 3447 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.889688 3353 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:17.027709 3534 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.028489 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:17.029106 32602 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:17.029680 3353 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.030431 3534 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.033176 3203 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:17.033977 3533 log.cc:924] Time spent T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260806 04:01:16.917834 3172 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.034617 32605 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
W20260806 04:01:16.897328 32738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1628) took 1705 ms (client timeout 1999 ms). Trace:
W20260806 04:01:17.037082 32741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1623) took 1876 ms (client timeout 2220 ms). Trace:
W20260806 04:01:17.037941 32615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 1943) took 2506 ms (client timeout 1526 ms). Trace:
I20260806 04:01:17.038901 1136 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:17.039314 1189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 619) took 2401 ms (client timeout 2444 ms). Trace:
W20260806 04:01:17.037346 32738 rpcz_store.cc:269] 0806 04:01:15.192098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.479295 (+287197us) service_pool.cc:227] Handling call
0806 04:01:16.897074 (+1417779us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54000a6ed4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:16.897241 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2061696256}
I20260806 04:01:17.040596 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.041536 32738 raft_consensus.cc:2396] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:01:17.004988 32609 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:16.934155 3319 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.049443 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:16.988710 32682 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.752s user 0.004s sys 0.019s
W20260806 04:01:17.039690 1189 rpcz_store.cc:269] 0806 04:01:14.638255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.641164 (+ 2909us) service_pool.cc:227] Handling call
0806 04:01:17.039118 (+2397954us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54006405d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:17.039232 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4150116096}
W20260806 04:01:17.050266 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 843320us
W20260806 04:01:16.998927 3524 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260806 04:01:17.012743 1188 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:17.015755 32749 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:17.014806 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.017457 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1529) took 2092 ms. Trace:
I20260806 04:01:17.018955 1192 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:17.020016 32428 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:17.016402 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:17.021629 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.205s user 0.016s sys 0.045s
W20260806 04:01:17.012130 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 135214us
W20260806 04:01:17.062733 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.24613e+06us
I20260806 04:01:17.064498 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:16.989543 3303 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:17.065337 3353 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:17.066555 3303 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.067466 3353 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:17.068320 3218 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:17.068809 3218 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.056313 32475 rpcz_store.cc:276] 0806 04:01:14.924859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.926273 (+ 1414us) service_pool.cc:227] Handling call
0806 04:01:14.940673 (+ 14400us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:14.940894 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:01:15.643790 (+702896us) log.cc:842] Serialized 14 byte log entry
0806 04:01:15.705344 (+ 61554us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b1800371c80. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb57b 00007fd16d25e88a
0806 04:01:17.017083 (+1311739us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:17.017140 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:17.017192 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:17.017272 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:17.017396 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28505,"prepare.queue_time_us":643013,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":6186,"raft.queue_time_us":3335,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5855,"spinlock_wait_cycles":24743680,"thread_start_us":1997158,"threads_started":2,"wal-append.queue_time_us":1274436}
I20260806 04:01:17.023656 3379 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:16.982852 32683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.743s user 0.010s sys 0.012s
I20260806 04:01:17.023442 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:17.022176 3415 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.025696 3283 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:16.898591 3477 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.017340 3462 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:17.042688 1136 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:17.007208 3538 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:17.038311 32615 rpcz_store.cc:269] 0806 04:01:14.531080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.531635 (+ 555us) service_pool.cc:227] Handling call
0806 04:01:17.037731 (+2506096us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b5400305f54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:17.037848 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5098592640}
W20260806 04:01:17.043534 32741 rpcz_store.cc:269] 0806 04:01:15.161076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.161237 (+ 161us) service_pool.cc:227] Handling call
0806 04:01:17.036909 (+1875672us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54003a4dd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:17.037026 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4205963136}
I20260806 04:01:17.047417 3496 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:17.049350 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:16.887019 3413 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.053102 32682 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:17.068784 32553 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:17.076462 1136 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:17.048966 3367 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:17.013339 3434 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.012833 3054 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:17.059160 3471 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.059813 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:17.060453 3178 tablet_bootstrap.cc:492] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Bootstrap complete.
I20260806 04:01:17.031976 3261 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.060874 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:17.103062 32746 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:17.073341 1190 raft_consensus.cc:2354] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:17.104717 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:17.071149 3218 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:17.081892 3542 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260806 04:01:17.074184 3477 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.083565 32553 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:17.084249 1187 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:17.021231 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.079115 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.085863 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:17.053884 32609 raft_consensus.cc:2379] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:17.085876 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 65917.9us
I20260806 04:01:17.071370 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.077114 1137 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:17.094592 32471 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:17.074435 3462 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:17.100508 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.071141 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:17.075773 3197 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.102340 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.074162 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:16.989604 3519 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:17.076249 3502 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:17.404012 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:17.082909 3283 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:17.326776 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:17.224444 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.097222 3203 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:17.223778 3178 ts_tablet_manager.cc:1397] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Time spent bootstrapping tablet: real 4.374s user 0.049s sys 0.088s
I20260806 04:01:17.094632 3504 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.106382 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.095467 3200 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.395964 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.188217 3452 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:17.071671 3379 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.102068 3054 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:17.223186 1183 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:17.100546 3134 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:17.397882 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 320822us
I20260806 04:01:17.097453 3538 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.217665 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.096868 3274 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:17.400547 32612 raft_consensus.cc:2379] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:17.400642 3519 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.106766 3367 raft_consensus.cc:686] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:17.317098 32602 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:17.174981 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:17.203459 32474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1523) took 2339 ms (client timeout 1837 ms). Trace:
I20260806 04:01:17.402838 1189 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:17.078227 3292 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:17.071332 3215 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:17.388420 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1726) took 1242 ms. Trace:
I20260806 04:01:17.487969 32469 rpcz_store.cc:276] 0806 04:01:16.146241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:16.146411 (+ 170us) service_pool.cc:227] Handling call
0806 04:01:16.146758 (+ 347us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:16.826576 (+679818us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:17.388135 (+561559us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b540003d254. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:17.388346 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1562125184}
I20260806 04:01:17.411686 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.412346 32477 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:01:17.415097 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.415599 32428 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:17.416306 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 332802us
I20260806 04:01:17.216409 32470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1535) took 1807 ms. Trace:
I20260806 04:01:17.327361 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.393824 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:17.194728 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 359563us
I20260806 04:01:17.490666 32470 rpcz_store.cc:276] 0806 04:01:15.409353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.548764 (+139411us) service_pool.cc:227] Handling call
0806 04:01:15.549137 (+ 373us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:15.549486 (+ 349us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:15.549514 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:15.549952 (+ 438us) log.cc:842] Serialized 26 byte log entry
0806 04:01:16.461125 (+911173us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:16.461162 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:16.461220 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:17.216195 (+754975us) raft_consensus.cc:1671] finished
0806 04:01:17.216216 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:17.216346 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11215,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":6431,"thread_start_us":894463,"threads_started":1,"wal-append.queue_time_us":1328618}
I20260806 04:01:17.322846 32746 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:17.418037 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:17.493281 32609 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:17.420755 32473 raft_consensus.cc:2379] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:17.381758 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 729) took 1562 ms. Trace:
I20260806 04:01:17.095428 3496 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:17.097182 3303 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 04:01:17.507680 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 92142.4us
I20260806 04:01:17.103006 3413 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.425503 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:17.417383 3532 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.015s sys 0.029s
I20260806 04:01:17.100782 1184 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:17.432346 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:17.473261 3538 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.435976 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:17.521768 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 86821.5us
I20260806 04:01:17.436636 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.414065 3344 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:17.438987 32682 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:17.070567 3449 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.424012 3477 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:17.420533 3188 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:17.424532 3283 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:17.470628 32615 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:17.534070 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:17.534900 32466 raft_consensus.cc:2354] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:17.484462 3172 raft_consensus.cc:2741] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 04:01:17.420293 3203 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:17.174728 3440 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.266s
I20260806 04:01:17.473979 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.474673 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:17.435143 3200 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:17.484738 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 736) took 1169 ms. Trace:
I20260806 04:01:17.591475 1189 rpcz_store.cc:276] 0806 04:01:16.315310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.054294 (+738984us) service_pool.cc:227] Handling call
0806 04:01:17.484673 (+430379us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:17.484489 32474 rpcz_store.cc:269] 0806 04:01:14.837497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.838197 (+ 700us) service_pool.cc:227] Handling call
0806 04:01:17.176830 (+2338633us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400012f54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:17.176993 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3492814592}
I20260806 04:01:17.507817 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.469866 3274 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.509234 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:17.496440 1188 rpcz_store.cc:276] 0806 04:01:15.810446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.812231 (+ 1785us) service_pool.cc:227] Handling call
0806 04:01:16.114438 (+302207us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:17.372432 (+1257994us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b54006342d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:17.372558 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1962194176}
I20260806 04:01:17.602301 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.491595 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:17.506175 32477 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:17.512823 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:17.603303 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 111783us
I20260806 04:01:17.469672 32737 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:17.612772 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.612591 3200 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.613454 32737 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:17.614358 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:17.585330 3344 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:17.615393 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:17.512694 3215 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.616021 32474 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:01:17.429843 3452 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.617516 3215 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.618685 3452 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.513403 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.520277 32746 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:17.620481 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.621699 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.622840 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.623190 32746 raft_consensus.cc:1226] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:17.513101 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.624354 32469 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:17.585153 3477 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.625207 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.521397 1186 raft_consensus.cc:2379] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:17.625857 1188 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:17.532191 3502 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260806 04:01:17.626727 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.521416 32605 raft_consensus.cc:2354] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:17.628652 3449 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.629402 1185 raft_consensus.cc:2354] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:17.415796 3197 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:17.631076 3140 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.438007 3134 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.543623 1137 leader_election.cc:398] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:17.641598 1137 leader_election.cc:401] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 04:01:17.420539 3319 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:17.523000 32609 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
W20260806 04:01:17.517000 3532 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.682s user 0.015s sys 0.029s
I20260806 04:01:17.487185 3292 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:17.485419 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.588575 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.593215 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.421622 32471 raft_consensus.cc:2354] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:17.516738 3437 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:17.512677 3303 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:17.518644 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1629) took 2326 ms. Trace:
I20260806 04:01:17.607666 3551 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:17.595113 3203 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:17.608585 3565 log.cc:924] Time spent T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260806 04:01:17.472458 3504 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260806 04:01:17.608863 3531 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:17.610144 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:17.611205 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.611763 3471 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:17.638082 1136 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:17.627885 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:17.623898 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.642508 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.643170 3230 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:17.614830 3200 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.654970 32748 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:17.657968 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:17.659956 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 154085us
I20260806 04:01:17.586751 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:17.661731 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.662820 32608 raft_consensus.cc:2379] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:17.626029 3477 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.665413 32477 raft_consensus.cc:2354] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:17.675626 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:17.648954 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 105493us
I20260806 04:01:17.678063 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:17.678737 1137 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:17.655889 3545 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:17.653895 3552 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:17.658059 3437 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:17.657557 3303 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:17.673949 32683 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:17.593603 3188 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:17.681677 32749 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260806 04:01:17.682263 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 78337.3us
I20260806 04:01:17.685086 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:17.701066 32477 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 3
I20260806 04:01:17.650977 3319 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.701848 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.687336 1137 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:17.694458 32470 raft_consensus.cc:1261] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:01:17.694832 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.614884 3367 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.697734 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.667537 3551 raft_consensus.cc:686] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:17.667027 3471 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:17.698431 32474 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:17.699487 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.706140 32683 leader_election.cc:398] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:17.718088 32683 leader_election.cc:401] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:01:17.707165 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.676412 3218 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:17.716046 32477 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:17.718091 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:17.720252 1185 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 730) took 1909 ms (client timeout 1526 ms). Trace:
W20260806 04:01:17.720644 1185 rpcz_store.cc:269] 0806 04:01:15.810734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.812313 (+ 1579us) service_pool.cc:227] Handling call
0806 04:01:17.720057 (+1907744us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400397e54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:17.720191 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3219429760}
I20260806 04:01:17.721695 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.722301 1185 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:17.715001 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.726881 3413 consensus_queue.cc:1046] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:17.727463 3134 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.727943 3413 raft_consensus.cc:3032] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:17.728351 3413 raft_consensus.cc:729] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:17.712204 3545 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:17.729118 3413 consensus_queue.cc:260] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.730592 3413 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:17.733666 3178 raft_consensus.cc:348] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:17.734508 3178 raft_consensus.cc:729] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Initialized, Role: LEARNER
I20260806 04:01:17.699538 3437 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:17.737463 3188 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.699775 3477 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:17.740281 3477 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:17.740674 3477 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:17.745456 3344 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:17.724375 3568 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:17.747088 3568 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:17.747412 3568 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:17.748240 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:17.748597 32743 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260806 04:01:17.749020 32743 pending_rounds.cc:76] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Aborting all ops after (but not including) 4
I20260806 04:01:17.749221 32743 pending_rounds.cc:97] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260806 04:01:17.749488 32743 raft_consensus.cc:2866] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:17.752542 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.753230 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:17.753847 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:17.754602 1182 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:17.755239 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:17.755396 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.755990 3218 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:17.756870 3452 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:17.756506 3218 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.757333 3452 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:17.757701 3452 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:17.679271 3230 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:17.758746 3218 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.759982 3230 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.759891 3218 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:17.671553 3440 consensus_queue.cc:1046] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260806 04:01:17.763288 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.766566 1185 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:17.760718 3218 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:17.768384 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:17.770550 1137 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:17.771595 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 69253.1us
I20260806 04:01:17.773488 3551 consensus_queue.cc:237] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.774061 3188 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:17.774753 3188 consensus_queue.cc:1035] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:17.776504 3344 raft_consensus.cc:3032] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:17.725966 32612 raft_consensus.cc:2379] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:17.707849 3054 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.777405 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:17.724409 3471 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:17.781533 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:17.782998 1137 leader_election.cc:398] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:17.783263 1137 leader_election.cc:401] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 04:01:17.749327 3319 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:17.785419 1185 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:17.785419 1189 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:17.786348 3283 leader_election.cc:288] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:17.788924 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 735) took 1509 ms. Trace:
I20260806 04:01:17.760733 3230 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:17.789543 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 339) took 1570 ms. Trace:
I20260806 04:01:17.778456 3218 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:17.791241 1189 rpcz_store.cc:276] 0806 04:01:16.279300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.784871 (+1505571us) service_pool.cc:227] Handling call
0806 04:01:17.785136 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:17.788723 (+ 3587us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:17.788878 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35840}
I20260806 04:01:17.788904 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 702) took 1936 ms. Trace:
I20260806 04:01:17.776949 3344 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:17.793942 32739 rpcz_store.cc:276] 0806 04:01:16.219078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:16.580035 (+360957us) service_pool.cc:227] Handling call
0806 04:01:16.580422 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:17.002393 (+421971us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:17.002422 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:17.002457 (+ 35us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:17.002494 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:17.789328 (+786834us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b54001f3954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:17.789375 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:17.789487 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":969912704}
I20260806 04:01:17.700052 3569 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.105s
W20260806 04:01:17.791182 1192 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 704) took 1887 ms (client timeout 1721 ms). Trace:
I20260806 04:01:17.784255 3502 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:17.787358 3496 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:17.797447 1185 rpcz_store.cc:276] 0806 04:01:15.852188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.784818 (+1932630us) service_pool.cc:227] Handling call
0806 04:01:17.785104 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:17.788678 (+ 3574us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:17.788854 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:17.797933 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 358974us
W20260806 04:01:17.799316 1192 rpcz_store.cc:269] 0806 04:01:15.904134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:16.252515 (+348381us) service_pool.cc:227] Handling call
0806 04:01:17.791014 (+1538499us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b540062edd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:17.791112 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1294662912}
I20260806 04:01:17.827426 1192 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 04:01:17.802796 3274 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:17.835052 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 52180.3us
I20260806 04:01:17.836143 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:17.735175 3178 consensus_queue.cc:260] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:17.838315 3496 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:17.839695 3344 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:17.840982 3344 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:17.854789 32682 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:17.833966 3437 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:17.909502 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 73441.8us
W20260806 04:01:17.920692 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 409728us
I20260806 04:01:17.797003 3230 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:01:17.933326 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 220079us
I20260806 04:01:17.656690 32740 rpcz_store.cc:276] 0806 04:01:15.192367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.993179 (+800812us) service_pool.cc:227] Handling call
0806 04:01:16.026624 (+ 33445us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:16.098378 (+ 71754us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:16.098413 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:16.098936 (+ 523us) log.cc:842] Serialized 26 byte log entry
0806 04:01:16.362279 (+263343us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:16.403196 (+ 40917us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:16.427993 (+ 24797us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:17.483406 (+1055413us) raft_consensus.cc:1671] finished
0806 04:01:17.485047 (+ 1641us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:17.516117 (+ 31070us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62863,"raft.queue_time_us":57671,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":195126,"threads_started":1,"wal-append.queue_time_us":875919}
I20260806 04:01:17.888617 3519 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:17.932958 3545 raft_consensus.cc:686] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:17.919480 3197 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:01:17.948522 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 175980us
I20260806 04:01:17.950912 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.953514 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.954133 32473 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
W20260806 04:01:17.943224 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 65563.6us
I20260806 04:01:17.945572 1137 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:17.947525 1189 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:17.958384 32683 leader_election.cc:302] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:17.812654 3203 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:17.958763 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.988555 32478 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:17.991238 32427 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:17.966187 32428 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:17.994938 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:18.001123 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:18.002893 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 44636.1us
I20260806 04:01:17.996870 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:17.820271 3292 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:18.006076 32473 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:18.006516 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:18.007485 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 400378us
I20260806 04:01:18.007695 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:18.011860 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:18.014009 32473 raft_consensus.cc:2379] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:18.015421 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:18.020771 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:18.010697 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 100171us
I20260806 04:01:18.022706 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.023444 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.023748 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:18.024133 32427 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.024571 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.025180 32610 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:18.023536 32609 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:18.025928 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:18.025954 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.026710 32474 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:18.027128 32475 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:18.027343 32610 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:18.028043 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.029321 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.028262 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.030908 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:18.028683 32603 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:18.031971 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.025167 1137 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:18.032648 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:18.033345 32603 raft_consensus.cc:2354] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:18.033555 1137 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.007962 3054 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:18.041646 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.040970 32614 raft_consensus.cc:1226] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:18.048983 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:18.049798 32466 raft_consensus.cc:2379] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:18.021878 3448 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:18.068958 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:18.070160 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 66644.5us
W20260806 04:01:18.073345 32553 connection.cc:232] RPC call timeout handler was delayed by -1.7606976928655058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
I20260806 04:01:18.026574 3562 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:18.085484 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:18.071081 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.399340 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:18.250942 32469 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.077s sys 0.150s
I20260806 04:01:18.367167 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.280194 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.400673 32746 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:18.322880 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:18.404445 32473 raft_consensus.cc:2379] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:18.402906 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:18.268800 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 246664us
I20260806 04:01:18.057613 3569 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 04:01:18.400348 32469 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.754s user 0.083s sys 0.157s
I20260806 04:01:18.275241 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.432068 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.106071 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:18.433300 1136 leader_election.cc:398] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:18.297154 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.434252 1136 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.435360 32741 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:18.400122 32466 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:18.282277 3178 ts_tablet_manager.cc:1428] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Time spent starting tablet: real 0.858s user 0.010s sys 0.007s
I20260806 04:01:18.073793 3448 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:18.451956 3178 ts_tablet_manager.cc:1460] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Tablet startup took 1031ms
I20260806 04:01:18.091969 3438 consensus_queue.cc:1046] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.327s
I20260806 04:01:18.037930 3586 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:18.090726 3545 consensus_queue.cc:237] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:18.090893 3054 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:18.422432 32610 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:18.452692 3576 raft_consensus.cc:3032] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:18.453279 3576 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:18.423732 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.423753 3603 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:18.424993 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:18.426030 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:18.403618 32553 connection.cc:232] RPC call timeout handler was delayed by -1.3610976929776371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260806 04:01:18.427232 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.464003 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:18.428867 3598 log.cc:924] Time spent T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.275s user 0.000s sys 0.000s
W20260806 04:01:18.345997 1190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 733) took 2534 ms (client timeout 1613 ms). Trace:
W20260806 04:01:18.390450 3568 consensus_meta.cc:287] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.643s user 0.009s sys 0.000s
I20260806 04:01:18.437229 3367 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:18.345932 3538 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:18.295717 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:18.441020 32471 raft_consensus.cc:2379] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:18.441270 3203 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:18.444160 3496 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.605s user 0.009s sys 0.000s
I20260806 04:01:18.447829 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.448626 32605 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:01:18.451184 3292 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:18.452261 3448 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:18.455653 32609 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
W20260806 04:01:18.455655 32553 connection.cc:232] RPC call timeout handler was delayed by -1.343296546023339s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.192s for a response
W20260806 04:01:18.503844 32553 connection.cc:232] RPC call timeout handler was delayed by -1.1621976930182427s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
W20260806 04:01:18.504215 32553 connection.cc:232] RPC call timeout handler was delayed by -1.104701038915664s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20260806 04:01:18.455233 3437 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:18.423105 3471 consensus_meta.cc:287] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.633s user 0.006s sys 0.000s
I20260806 04:01:18.463793 32603 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:18.516029 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 496057us
I20260806 04:01:18.517611 32682 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:18.516273 3200 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.036s sys 0.060s
W20260806 04:01:18.528626 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 504541us
I20260806 04:01:18.529435 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.531512 32427 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.518596 3471 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:18.467706 1190 rpcz_store.cc:269] 0806 04:01:15.811633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.811779 (+ 146us) service_pool.cc:227] Handling call
0806 04:01:18.345771 (+2533992us) spinlock_profiling.cc:243] Waited 903 ms on lock 0x7b54005cdfd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:18.345923 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2077531520}
I20260806 04:01:18.469581 3603 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:18.500818 3367 raft_consensus.cc:686] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:18.455768 3586 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:18.471356 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.159263 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.552126 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
W20260806 04:01:18.528240 3452 consensus_meta.cc:287] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.770s user 0.009s sys 0.002s
I20260806 04:01:18.553239 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 483) took 1032 ms. Trace:
I20260806 04:01:18.553742 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1746) took 1142 ms. Trace:
W20260806 04:01:18.504506 32553 connection.cc:232] RPC call timeout handler was delayed by -1.0561010390520096s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
I20260806 04:01:18.455186 32614 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:18.553565 32604 rpcz_store.cc:276] 0806 04:01:17.521104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.521355 (+ 251us) service_pool.cc:227] Handling call
0806 04:01:18.552657 (+1031302us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b58001e5798. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:18.552831 (+ 174us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:18.552976 (+ 145us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:18.553007 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:18.553040 (+ 33us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:18.553066 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.553108 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:18.553190 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8707456}
W20260806 04:01:18.541911 3200 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s user 0.036s sys 0.060s
I20260806 04:01:18.561303 32615 raft_consensus.cc:2379] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:18.516675 3437 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:18.556941 3438 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:18.516021 3413 consensus_meta.cc:287] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.785s user 0.005s sys 0.006s
W20260806 04:01:18.513134 3303 consensus_meta.cc:287] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.801s user 0.009s sys 0.005s
I20260806 04:01:18.544911 32682 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:18.574023 32682 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:18.469551 3054 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:18.454532 3576 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.557353 3531 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:18.152945 3562 raft_consensus.cc:686] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:18.548813 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.513491 3496 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:18.154675 3582 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:18.596649 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.598489 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.461370 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.511334 3230 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:18.482038 3568 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.512643 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:18.296774 3588 log.cc:924] Time spent T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.205s user 0.000s sys 0.000s
W20260806 04:01:18.563288 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 124840us
I20260806 04:01:18.537585 3292 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:18.542558 3283 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:18.557026 3586 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:18.571422 3437 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:18.556820 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.561053 3452 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:18.452288 3178 ts_tablet_manager.cc:1462] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Trace:
0806 04:01:08.732574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:08.733837 (+ 1263us) service_pool.cc:227] Handling call
0806 04:01:09.223083 (+489246us) ts_tablet_manager.cc:928] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Initiating tablet copy from peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
0806 04:01:10.955340 (+1732257us) tablet_copy_client.cc:361] Tablet copy session begun
0806 04:01:11.596069 (+640729us) tablet_metadata.cc:641] Metadata flushed
0806 04:01:11.624176 (+ 28107us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 04:01:12.225152 (+600976us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 04:01:12.261554 (+ 36402us) tablet_replica.cc:205] Creating consensus instance
0806 04:01:12.302448 (+ 40894us) ts_tablet_manager.cc:983] Replica registered.
0806 04:01:12.302515 (+ 67us) inbound_call.cc:173] Queueing success response
0806 04:01:12.849972 (+547457us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0806 04:01:17.423302 (+4573330us) ts_tablet_manager.cc:1429] Starting tablet replica
0806 04:01:17.731652 (+308350us) tablet_replica.cc:259] Starting instrumentation
0806 04:01:17.732253 (+ 601us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":2298,"mutex_wait_us":23573,"raft.queue_time_us":3405,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":5415,"tablet-copy.queue_time_us":36927,"tablet-download-pool.queue_time_us":26355,"tablet-download-pool.run_cpu_time_us":1548,"tablet-download-pool.run_wall_time_us":78654,"thread_start_us":52220,"threads_started":2}
I20260806 04:01:18.601747 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.603752 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:18.551456 3603 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.613240 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.591202 3593 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:18.593351 3572 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:18.611827 3462 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:18.597793 3576 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:18.617128 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:18.629570 3562 consensus_queue.cc:237] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.594710 3519 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260806 04:01:18.619565 1182 raft_consensus.cc:2379] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:18.598045 3582 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:18.634516 3582 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:18.635071 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 644) took 1261 ms. Trace:
I20260806 04:01:18.622645 3471 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:18.588223 3200 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 04:01:18.571668 3367 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.626150 3603 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:18.600519 3569 raft_consensus.cc:686] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:18.589169 3303 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.592445 3531 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:18.628724 3601 log.cc:924] Time spent T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260806 04:01:18.618328 3572 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:18.629158 1186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 707) took 1632 ms (client timeout 1686 ms). Trace:
I20260806 04:01:18.629510 3437 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:18.652455 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.675231 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.676000 1190 raft_consensus.cc:2354] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:18.676734 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.677769 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.678715 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.679622 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:18.660782 3477 consensus_meta.cc:287] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.920s user 0.009s sys 0.001s
W20260806 04:01:18.664798 3344 consensus_meta.cc:287] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.823s user 0.005s sys 0.005s
I20260806 04:01:18.672883 3572 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:18.694160 3531 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:18.695111 3569 consensus_queue.cc:237] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.699344 3568 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:18.702190 3552 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.003s user 0.006s sys 0.003s
I20260806 04:01:18.702637 3552 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3.
I20260806 04:01:18.554142 32469 rpcz_store.cc:276] 0806 04:01:17.411459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.641637 (+230178us) service_pool.cc:227] Handling call
0806 04:01:17.641877 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:17.642047 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:01:18.430858 (+788811us) log.cc:842] Serialized 14 byte log entry
0806 04:01:18.553400 (+122542us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:18.553452 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:18.553507 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.553572 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:18.553683 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9572,"prepare.queue_time_us":399020,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"thread_start_us":897225,"threads_started":2}
I20260806 04:01:18.555799 32553 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:18.748126 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 558) took 1668 ms. Trace:
I20260806 04:01:18.750836 32602 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 04:01:18.757809 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 201937us
W20260806 04:01:18.666267 1186 rpcz_store.cc:269] 0806 04:01:16.996709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.393554 (+396845us) service_pool.cc:227] Handling call
0806 04:01:18.628967 (+1235413us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b540067a554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:18.629097 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57162496}
I20260806 04:01:18.760002 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 639) took 1476 ms. Trace:
I20260806 04:01:18.761053 32748 raft_consensus.cc:1226] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 04:01:18.614156 3292 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:18.660038 1137 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.004s sys 0.004s
I20260806 04:01:18.653815 1188 raft_consensus.cc:1226] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
W20260806 04:01:18.789975 1137 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.718s user 0.004s sys 0.004s
I20260806 04:01:18.791666 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 734) took 2531 ms. Trace:
I20260806 04:01:18.709803 3582 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.687397 3477 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.773644 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42618 (request call id 481) took 1264 ms. Trace:
I20260806 04:01:18.776984 3521 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 04:01:18.791895 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 758337us
I20260806 04:01:18.778394 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 732) took 2966 ms. Trace:
I20260806 04:01:18.806732 1183 rpcz_store.cc:276] 0806 04:01:15.811334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.829739 (+ 18405us) service_pool.cc:227] Handling call
0806 04:01:16.225260 (+395521us) spinlock_profiling.cc:243] Waited 6.31 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:16.239383 (+ 14123us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:17.430293 (+1190910us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.777730 (+1347437us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b54005bac54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:18.778161 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1470160384}
I20260806 04:01:18.785631 32738 raft_consensus.cc:2396] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:01:18.785822 3452 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:18.816913 32470 raft_consensus.cc:1226] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:18.788650 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:18.635473 1182 rpcz_store.cc:276] 0806 04:01:17.373204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.755145 (+381941us) service_pool.cc:227] Handling call
0806 04:01:18.634865 (+879720us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54007a1a54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:18.634992 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1980483328}
I20260806 04:01:18.760555 1192 rpcz_store.cc:276] 0806 04:01:17.283340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.829114 (+545774us) service_pool.cc:227] Handling call
0806 04:01:17.836357 (+ 7243us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:17.836866 (+ 509us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:17.836895 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:17.876702 (+ 39807us) log.cc:842] Serialized 26 byte log entry
0806 04:01:18.333992 (+457290us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:18.334033 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.334115 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:18.759272 (+425157us) raft_consensus.cc:1671] finished
0806 04:01:18.759319 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:18.759683 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1694,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"thread_start_us":343132,"threads_started":1,"wal-append.queue_time_us":590878}
W20260806 04:01:18.793401 32551 leader_election.cc:334] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 1.686s (SENT)
I20260806 04:01:18.776798 3590 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
W20260806 04:01:18.673956 3218 consensus_meta.cc:287] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.873s user 0.008s sys 0.001s
I20260806 04:01:18.821482 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:18.771740 3303 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:18.777083 32553 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.827332 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:18.828385 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 51564.7us
I20260806 04:01:18.829035 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 567) took 1407 ms. Trace:
I20260806 04:01:18.829447 32478 rpcz_store.cc:276] 0806 04:01:17.421061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.646800 (+225739us) service_pool.cc:227] Handling call
0806 04:01:17.663679 (+ 16879us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:17.998741 (+335062us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.828515 (+829774us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b5400368dd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:18.828859 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":746678528}
I20260806 04:01:18.831370 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.832432 3521 raft_consensus.cc:686] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:18.834173 3521 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.836472 3434 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:18.837018 32631 tablet_copy_service.cc:342] P dd7344d85a154539b7f8c484f2a76772: Request end of tablet copy session d4a238f055a44f459c62c224daa4d541-12e7971e809846ada28ebf1d6cea4837 received from {username='slave'} at 127.0.0.1:42618
I20260806 04:01:18.837682 32631 tablet_copy_service.cc:433] P dd7344d85a154539b7f8c484f2a76772: ending tablet copy session d4a238f055a44f459c62c224daa4d541-12e7971e809846ada28ebf1d6cea4837 on tablet 12e7971e809846ada28ebf1d6cea4837 with peer d4a238f055a44f459c62c224daa4d541
I20260806 04:01:18.810191 3292 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.841194 3292 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:18.842206 3477 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:18.844424 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:18.845731 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:18.845595 32321 catalog_manager.cc:5495] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 2, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 2 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:18.798996 1188 rpcz_store.cc:276] 0806 04:01:16.260430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.779687 (+1519257us) service_pool.cc:227] Handling call
0806 04:01:17.791924 (+ 12237us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:18.790112 (+998188us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.791135 (+ 1023us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54004efcd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:18.791443 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1977877376}
I20260806 04:01:18.812383 3582 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:18.785779 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:18.836774 1184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 728) took 3026 ms (client timeout 1822 ms). Trace:
I20260806 04:01:18.812104 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 739) took 1976 ms. Trace:
I20260806 04:01:18.850828 1186 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 1 LEARNER]: Advancing to term 2
I20260806 04:01:18.748677 32475 rpcz_store.cc:276] 0806 04:01:17.079912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.080026 (+ 114us) service_pool.cc:227] Handling call
0806 04:01:17.100895 (+ 20869us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:18.027510 (+926615us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.747417 (+719907us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54000492d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:18.747804 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2119418240}
I20260806 04:01:18.821696 3218 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:18.850898 1184 rpcz_store.cc:269] 0806 04:01:15.810118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:15.830306 (+ 20188us) service_pool.cc:227] Handling call
0806 04:01:18.836375 (+3006069us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b54004f88d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:18.836530 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1911004416}
I20260806 04:01:18.805749 32608 rpcz_store.cc:276] 0806 04:01:17.508789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.508973 (+ 184us) service_pool.cc:227] Handling call
0806 04:01:18.773003 (+1264030us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b540007f3d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:18.773360 (+ 357us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135113472}
I20260806 04:01:18.822012 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.798236 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2077) took 1933 ms. Trace:
I20260806 04:01:18.854848 32745 rpcz_store.cc:276] 0806 04:01:16.864673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:16.864802 (+ 129us) service_pool.cc:227] Handling call
0806 04:01:16.868007 (+ 3205us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:17.399247 (+531240us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:17.399275 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:17.399312 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:17.399338 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.797633 (+1398295us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b5400244354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:18.797711 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:18.798018 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1221313280}
I20260806 04:01:18.825141 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.854332 1189 rpcz_store.cc:276] 0806 04:01:16.835483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.798773 (+963290us) service_pool.cc:227] Handling call
0806 04:01:17.799047 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:17.968778 (+169731us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.811595 (+842817us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54004736d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:18.811909 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":286218496}
I20260806 04:01:18.859001 1190 raft_consensus.cc:1226] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:18.836937 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:18.828979 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:18.861053 1185 raft_consensus.cc:1226] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:18.875295 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 635) took 1770 ms. Trace:
I20260806 04:01:18.875725 1185 rpcz_store.cc:276] 0806 04:01:17.104751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.821474 (+716723us) service_pool.cc:227] Handling call
0806 04:01:17.821799 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:18.874161 (+1052362us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.874991 (+ 830us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540078e454. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:18.875215 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2378028800}
I20260806 04:01:18.854539 3218 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:18.877790 1182 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:18.879401 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:18.870656 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:18.873224 32324 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36880 (request call id 231) took 5538 ms. Trace:
I20260806 04:01:18.850632 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 642) took 1495 ms. Trace:
I20260806 04:01:18.885480 1188 rpcz_store.cc:276] 0806 04:01:17.355550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:18.849034 (+1493484us) service_pool.cc:227] Handling call
0806 04:01:18.849656 (+ 622us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:18.850087 (+ 431us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:18.850122 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:18.850189 (+ 67us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:18.850274 (+ 85us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.850356 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:18.850532 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:18.889714 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 645) took 1506 ms. Trace:
I20260806 04:01:18.890384 1184 rpcz_store.cc:276] 0806 04:01:17.383052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:18.853351 (+1470299us) service_pool.cc:227] Handling call
0806 04:01:18.853845 (+ 494us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:18.889168 (+ 35323us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:18.889221 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:18.889291 (+ 70us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:18.889379 (+ 88us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.889454 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:18.889609 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28994,"raft.queue_time_us":34293,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":208}
I20260806 04:01:18.893282 1184 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 04:01:18.892458 32321 catalog_manager.cc:5495] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:18.894237 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:18.894560 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 63390.9us
I20260806 04:01:18.893055 32745 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:18.896588 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 715) took 1278 ms. Trace:
I20260806 04:01:18.893622 1192 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:18.900661 1190 rpcz_store.cc:276] 0806 04:01:17.604882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:18.855253 (+1250371us) service_pool.cc:227] Handling call
0806 04:01:18.855638 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:18.883306 (+ 27668us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.883486 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:18.904160 32553 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.906251 32553 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.908787 32321 catalog_manager.cc:5495] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:18.909914 32602 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:18.910275 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:18.910583 3379 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:18.911358 3379 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:18.912114 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:18.913085 32604 raft_consensus.cc:2379] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:18.912226 3590 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:18.916468 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:18.916734 3200 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:18.917505 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:18.920543 3274 consensus_queue.cc:1046] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260806 04:01:18.925097 3437 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:18.926056 3437 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:18.929150 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:18.930576 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.930970 1187 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:18.931663 3590 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:18.930594 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:18.932570 32553 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.932973 1192 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:18.933672 3590 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:18.916157 3636 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:18.934639 3636 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.940186 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:18.940670 32553 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.941098 32602 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:18.946877 3215 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:18.947479 3218 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:18.948475 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:18.948675 3636 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:18.949326 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.950268 32604 raft_consensus.cc:2354] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:18.907698 3586 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.949864 3215 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.951530 32473 raft_consensus.cc:1226] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:18.953956 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:18.954326 3215 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:18.956998 32738 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:18.957636 3586 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:18.958214 3379 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:18.962253 32553 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:18.962932 3448 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:18.949976 32427 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:18.964880 3572 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.965613 3437 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:18.933568 1187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 716) took 1328 ms (client timeout 1653 ms). Trace:
I20260806 04:01:18.971074 1185 raft_consensus.cc:1226] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
W20260806 04:01:18.976586 1187 rpcz_store.cc:269] 0806 04:01:17.605166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:17.677830 (+ 72664us) service_pool.cc:227] Handling call
0806 04:01:18.933342 (+1255512us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54006342d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:18.933500 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2505003776}
I20260806 04:01:18.985006 3477 consensus_queue.cc:1046] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:18.985466 32749 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:18.986255 3477 raft_consensus.cc:3032] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:18.986651 3477 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:18.987546 3188 raft_consensus.cc:3032] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:18.987891 32741 raft_consensus.cc:2379] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:18.988114 3218 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:18.989123 3547 consensus_queue.cc:1046] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:18.990469 3547 raft_consensus.cc:3032] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:18.990904 3547 raft_consensus.cc:729] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:18.992622 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:18.995136 3188 raft_consensus.cc:729] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:18.995303 3367 consensus_queue.cc:1046] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:19.012672 3547 consensus_queue.cc:260] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:18.976397 3590 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:19.014094 3590 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:19.014392 3547 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.015399 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:19.016052 32604 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
W20260806 04:01:19.017261 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 67493.2us
I20260806 04:01:19.018167 32427 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.019567 32427 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.014963 3188 consensus_queue.cc:260] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.020738 3188 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.023016 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.023947 3174 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:19.014531 3590 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.027099 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.020812 1136 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.030769 3448 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:19.032114 3572 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:19.032783 3437 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:18.987581 3477 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.035458 3477 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:18.884503 32324 rpcz_store.cc:276] 0806 04:01:13.334586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:13.334742 (+ 156us) service_pool.cc:227] Handling call
0806 04:01:18.872921 (+5538179us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:15.060597 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:15.173509 (+112912us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:15.173563 (+ 54us) write_op.cc:435] Acquired schema lock
0806 04:01:15.173601 (+ 38us) tablet.cc:660] Decoding operations
0806 04:01:15.174138 (+ 537us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:15.174284 (+ 146us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:15.174331 (+ 47us) tablet.cc:683] Acquiring locks for 1 operations
0806 04:01:15.174632 (+ 301us) tablet.cc:699] Row locks acquired
0806 04:01:15.174669 (+ 37us) write_op.cc:260] PREPARE: finished
0806 04:01:15.174867 (+ 198us) write_op.cc:270] Start()
0806 04:01:15.175055 (+ 188us) write_op.cc:276] Timestamp: P: 1785988875174813 usec, L: 0
0806 04:01:15.175126 (+ 71us) op_driver.cc:331] REPLICATION: starting
0806 04:01:15.297570 (+122444us) log.cc:842] Serialized 539 byte log entry
0806 04:01:16.832864 (+1535294us) spinlock_profiling.cc:243] Waited 5.61 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:17.472453 (+639589us) op_driver.cc:444] REPLICATION: finished
0806 04:01:17.814471 (+342018us) write_op.cc:301] APPLY: starting
0806 04:01:17.814552 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:17.814663 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:17.814683 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:17.814937 (+ 254us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:17.815140 (+ 203us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 04:01:17.817275 (+ 2135us) write_op.cc:312] APPLY: finished
0806 04:01:17.823432 (+ 6157us) log.cc:842] Serialized 23 byte log entry
0806 04:01:17.843559 (+ 20127us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:17.943279 (+ 99720us) write_op.cc:454] Released schema lock
0806 04:01:17.943329 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:18.625596 (+682267us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b18003151c0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a2f9f96 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd170b9b361 00007fd170c6c254 00007fd170c6586b 00007fd170c64fd1
0806 04:01:18.626814 (+ 1218us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b5400002680. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd170b9b361 00007fd170c6c254 00007fd170c6586b 00007fd170c64fd1 00007fd170c68dd1 00007fd170c68d89 00007fd170c68d19 00007fd170c68ce1 00007fd170c67fdd 00007fd173590204 00007fd173590039
0806 04:01:18.626921 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:18.626946 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":460520,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":329213,"apply.run_cpu_time_us":24605,"apply.run_wall_time_us":812720,"num_ops":1,"prepare.queue_time_us":481411,"prepare.run_cpu_time_us":48642,"prepare.run_wall_time_us":1873928,"raft.queue_time_us":591588,"raft.run_cpu_time_us":12920,"raft.run_wall_time_us":520698,"replication_time_us":2265077,"spinlock_wait_cycles":94569728,"thread_start_us":3001856,"threads_started":4,"wal-append.queue_time_us":1214962,"wal-append.run_cpu_time_us":54480,"wal-append.run_wall_time_us":2277922}]]}
I20260806 04:01:19.040170 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:19.040278 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:19.044250 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:19.044979 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:19.046851 32470 raft_consensus.cc:1261] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:01:19.048164 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:19.048921 32741 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:19.052379 3200 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:01:19.055359 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 93208.7us
I20260806 04:01:19.061127 3449 raft_consensus.cc:3032] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:19.061556 3449 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:19.062526 3449 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.063483 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:19.065131 3367 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:19.065677 32741 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:19.069886 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:19.070680 32739 raft_consensus.cc:2379] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:19.052639 3636 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.074085 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 651) took 1041 ms. Trace:
I20260806 04:01:19.076342 1188 rpcz_store.cc:276] 0806 04:01:18.019084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:18.897727 (+878643us) service_pool.cc:227] Handling call
0806 04:01:18.975668 (+ 77941us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:19.056015 (+ 80347us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:19.056041 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:19.060239 (+ 4198us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:19.060278 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:19.060332 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:19.060438 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:19.077663 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 51044.3us
I20260806 04:01:19.077852 3647 consensus_queue.cc:1035] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 04:01:19.078706 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:19.083735 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 63141.2us
I20260806 04:01:19.065078 3449 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:19.087754 32602 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:19.088390 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.074452 3452 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 04:01:19.105252 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 85827.5us
I20260806 04:01:19.111697 3379 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:19.113360 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:19.096361 3452 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:19.125054 3590 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.116459 3379 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:19.118525 3452 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.132413 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.152745 32427 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.154259 32553 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.163720 32743 raft_consensus.cc:2379] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:19.162665 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:19.165277 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 58498.4us
I20260806 04:01:19.133070 3379 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.222465 32738 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:19.226644 3636 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:19.227301 3636 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:19.228468 3636 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:19.230710 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 65419.1us
I20260806 04:01:19.171823 32682 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:19.242951 1136 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
W20260806 04:01:19.257552 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 143598us
W20260806 04:01:19.250782 3659 log.cc:924] Time spent T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.121s user 0.003s sys 0.005s
I20260806 04:01:19.270880 1137 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:19.271261 32737 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
W20260806 04:01:19.272347 3662 log.cc:924] Time spent T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.071s user 0.002s sys 0.000s
W20260806 04:01:19.272547 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 54293us
I20260806 04:01:19.272747 3215 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:19.273197 3215 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:19.273428 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 159393us
I20260806 04:01:19.273933 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 643) took 1912 ms. Trace:
I20260806 04:01:19.274219 3261 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:19.274950 32682 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.276609 3274 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:19.276875 3215 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.281872 32741 raft_consensus.cc:3032] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:19.277004 3572 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:19.282262 32741 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:19.276727 3657 log.cc:924] Time spent T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
I20260806 04:01:19.288933 3200 consensus_queue.cc:1046] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:19.274880 3261 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.332554 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 720) took 1700 ms. Trace:
I20260806 04:01:19.289695 1136 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.277045 3274 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:19.292469 32682 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.335017 3274 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.274292 1186 rpcz_store.cc:276] 0806 04:01:17.362046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:18.849900 (+1487854us) service_pool.cc:227] Handling call
0806 04:01:18.850507 (+ 607us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:18.985024 (+134517us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:18.985058 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:18.985532 (+ 474us) log.cc:842] Serialized 26 byte log entry
0806 04:01:19.052220 (+ 66688us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:19.103038 (+ 50818us) log.cc:842] Serialized 14 byte log entry
0806 04:01:19.103353 (+ 315us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:19.103503 (+ 150us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:19.273718 (+170215us) raft_consensus.cc:1671] finished
0806 04:01:19.273734 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:19.273854 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1883,"prepare.queue_time_us":171107,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":3716,"raft.queue_time_us":3248,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":3093,"thread_start_us":113760,"threads_started":2,"wal-append.queue_time_us":99520}
W20260806 04:01:19.295514 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 177429us
I20260806 04:01:19.299348 3569 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:19.300992 32475 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260806 04:01:19.302162 32473 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:19.308228 3647 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:19.313206 32470 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:19.256361 3434 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.345192 3647 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.326308 32428 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:19.328474 3590 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:19.331043 3200 raft_consensus.cc:3032] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:19.346700 3477 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:19.347107 3200 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:19.347649 3647 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:19.296293 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.337584 32475 pending_rounds.cc:76] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 4
I20260806 04:01:19.342464 3562 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:19.348066 3200 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.361212 3292 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:19.351105 3531 consensus_queue.cc:1046] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:19.352322 3569 raft_consensus.cc:686] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:19.352474 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:19.353524 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 63927.3us
I20260806 04:01:19.331113 3215 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.332947 1189 rpcz_store.cc:276] 0806 04:01:17.632627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:18.861206 (+1228579us) service_pool.cc:227] Handling call
0806 04:01:18.861698 (+ 492us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:18.893931 (+ 32233us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:18.893964 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:18.894543 (+ 579us) log.cc:842] Serialized 26 byte log entry
0806 04:01:18.974392 (+ 79849us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:01:18.974430 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:18.974491 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:19.332258 (+357767us) raft_consensus.cc:1671] finished
0806 04:01:19.332281 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:19.332489 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31482,"raft.queue_time_us":32739,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":24230,"thread_start_us":79376,"threads_started":1,"wal-append.queue_time_us":181354}
I20260806 04:01:19.333011 3261 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:19.359650 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.359706 3639 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:19.360900 32746 raft_consensus.cc:1226] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:19.357414 32475 pending_rounds.cc:97] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260806 04:01:19.366693 32475 raft_consensus.cc:2866] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:19.361692 3292 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:19.361382 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 69043.7us
I20260806 04:01:19.360944 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:19.376185 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 50207.6us
I20260806 04:01:19.375458 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:19.389132 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 93825.7us
I20260806 04:01:19.389397 32475 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260806 04:01:19.377830 3678 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.081s user 0.002s sys 0.000s
I20260806 04:01:19.373117 3292 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.378856 32741 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.396137 32745 raft_consensus.cc:1226] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:19.390084 3367 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260806 04:01:19.380501 3437 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.384011 3582 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:19.384526 3200 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:19.394960 32467 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:19.409564 32741 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.406077 3521 raft_consensus.cc:3032] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:19.379760 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:19.425122 32682 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.407210 3582 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:19.428856 32553 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.435209 32428 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:19.439549 32749 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:19.411361 3521 raft_consensus.cc:729] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:19.410598 3531 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:19.447127 3531 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.449366 3531 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:19.451278 3215 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:19.433784 3582 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.455205 3569 consensus_queue.cc:237] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.444134 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:19.458638 32466 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:19.425675 32740 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
W20260806 04:01:19.473276 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 83409.5us
I20260806 04:01:19.438766 3434 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:01:19.476488 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 51753.8us
I20260806 04:01:19.480130 1137 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:19.457944 3647 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:19.483405 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 85216.8us
I20260806 04:01:19.466917 3215 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:19.487759 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.472728 3437 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:19.492219 32553 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.481389 3188 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:19.477156 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.497300 3452 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.493491 3218 raft_consensus.cc:2781] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:19.505589 32553 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.506480 32682 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:19.499935 3434 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:19.509792 32427 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.504237 3218 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:19.510695 3531 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:19.511523 3303 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:19.501636 3215 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.512674 3434 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:19.519598 3218 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.531129 32553 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.532162 32683 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:19.532979 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 55367.8us
I20260806 04:01:19.533085 3647 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:19.533565 3647 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:19.533910 3647 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.537696 3636 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:19.538123 3636 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:19.538480 3636 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.538717 3274 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.509914 3188 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:19.524967 3303 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:19.542052 3188 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:19.543079 32738 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:19.543987 32746 raft_consensus.cc:1226] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:19.541925 3303 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.479202 3379 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.521706 3292 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.546612 3647 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.500391 3521 consensus_queue.cc:260] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.526736 3434 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.555635 3521 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.556592 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:19.557708 3292 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:19.559458 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 73868.7us
I20260806 04:01:19.559600 32553 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.560351 32427 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.560963 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.561815 32682 leader_election.cc:398] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:19.561972 3590 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.561827 3172 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.562860 32682 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.563776 3188 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:19.565281 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 71836.7us
I20260806 04:01:19.567296 32737 raft_consensus.cc:1226] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:19.570602 32739 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:19.572305 3452 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:19.573804 3215 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:19.564422 3531 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.579516 3434 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.580837 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.582909 3434 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:19.583462 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:19.589639 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.591594 32427 leader_election.cc:398] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:19.591892 32427 leader_election.cc:401] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:19.592356 3603 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.592361 3690 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.596163 3477 consensus_queue.cc:1046] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:19.611299 3415 raft_consensus.cc:3032] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:19.612308 3572 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:19.612805 3572 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.596709 3274 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:19.551745 3582 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.613811 32474 raft_consensus.cc:1226] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:19.614717 3379 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:19.611752 3415 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:19.616819 3582 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:19.605003 32741 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 04:01:19.617269 3647 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:19.617817 3303 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.612352 32682 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.611339 32553 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.626291 3215 consensus_queue.cc:1046] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:19.629415 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:19.641462 32741 pending_rounds.cc:76] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Aborting all ops after (but not including) 5
I20260806 04:01:19.645622 32741 pending_rounds.cc:97] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 04:01:19.610993 3218 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.650156 32469 raft_consensus.cc:2379] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:19.651764 32741 raft_consensus.cc:2866] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:19.653621 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.659658 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.656418 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:19.665819 32478 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.643023 3437 raft_consensus.cc:686] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:19.662811 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 389870us
I20260806 04:01:19.593122 3636 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.698698 3303 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:19.695971 32428 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:19.699584 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:19.700537 3218 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:19.702266 3562 raft_consensus.cc:3032] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:19.702661 3637 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:19.705118 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.705297 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:19.653496 3586 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260806 04:01:19.708329 1186 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 04:01:19.709120 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 96900.1us
I20260806 04:01:19.709677 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:19.688937 3367 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.729171 32551 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:19.728969 32475 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.689568 3379 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:19.732864 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 161517us
I20260806 04:01:19.742004 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:19.723371 3603 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:19.726714 3562 raft_consensus.cc:729] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:19.761171 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 169708us
I20260806 04:01:19.764045 1187 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.736840 32737 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.757350 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:19.712488 3261 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20260806 04:01:19.771582 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 45458.3us
I20260806 04:01:19.772614 32551 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:19.740375 3200 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:19.777048 32427 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.784301 32553 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.789011 32682 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.790453 32682 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.791369 3274 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.793573 3636 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:19.794584 3562 consensus_queue.cc:260] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.795862 3562 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.796762 3379 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:19.797183 3379 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:19.797499 3379 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:19.800624 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:19.801388 32739 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:19.803439 3572 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.804553 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:19.758036 3434 raft_consensus.cc:2741] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 04:01:19.713018 3415 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.819293 3572 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:19.820562 32682 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.822026 32682 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.822172 32738 raft_consensus.cc:1226] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:19.822872 3637 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.823559 3637 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.824214 3637 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:19.826941 3415 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.828627 1184 raft_consensus.cc:2362] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3. Re-sending same reply.
I20260806 04:01:19.830672 3200 raft_consensus.cc:686] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:19.831692 3200 consensus_queue.cc:237] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.830698 3636 consensus_queue.cc:1046] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 04:01:19.834218 3636 raft_consensus.cc:3032] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:19.834640 3636 raft_consensus.cc:729] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:19.837989 3531 consensus_queue.cc:1046] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:19.840888 3590 consensus_queue.cc:1046] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:19.841465 32427 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.841712 3697 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:19.842536 3697 raft_consensus.cc:2781] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:19.842772 32427 leader_election.cc:302] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.845454 32475 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:19.847497 1137 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
W20260806 04:01:19.849857 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 65673.5us
I20260806 04:01:19.850335 32478 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:19.792029 3274 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:19.851732 1187 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:19.853550 32683 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:19.853767 32553 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.838701 3172 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260806 04:01:19.855126 32427 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:19.828702 3521 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:19.849080 3697 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:19.851712 3218 raft_consensus.cc:2781] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:19.877086 3283 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.877240 3303 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:19.878099 32682 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:19.878386 3647 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:19.879529 32682 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.859969 3704 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.880419 32738 raft_consensus.cc:1226] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:19.879873 3647 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:19.882085 3283 consensus_queue.cc:1046] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:19.883180 3283 raft_consensus.cc:3032] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:19.883507 3283 raft_consensus.cc:729] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:19.884563 3590 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.884374 3283 consensus_queue.cc:260] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.885313 3590 raft_consensus.cc:3032] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:19.835633 3636 consensus_queue.cc:260] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.887578 3283 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:19.910559 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 59089.1us
I20260806 04:01:19.864718 3367 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.864426 3274 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:19.918450 32737 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:19.920254 32738 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:19.885670 3590 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:19.910712 3647 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:19.886765 3697 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:19.932823 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:19.889165 3218 raft_consensus.cc:686] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:19.942422 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 103721us
I20260806 04:01:19.943694 32551 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:19.944907 3172 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:19.945416 3172 raft_consensus.cc:686] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:19.946373 3172 consensus_queue.cc:237] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.947067 3134 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260806 04:01:19.948843 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:19.864403 3437 consensus_queue.cc:237] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.950237 3711 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:19.950835 3711 raft_consensus.cc:686] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:19.951900 3711 consensus_queue.cc:237] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.952245 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:19.954280 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:19.955205 32602 raft_consensus.cc:2396] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260806 04:01:19.940353 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 60916us
I20260806 04:01:19.960426 3521 raft_consensus.cc:3032] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:19.960840 3521 raft_consensus.cc:729] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:19.961745 3521 consensus_queue.cc:260] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.962730 3521 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.966862 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:19.967535 1187 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:19.968652 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:19.969848 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:19.970384 3636 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:19.973074 3218 consensus_queue.cc:237] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.931238 3274 raft_consensus.cc:686] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:19.925529 3379 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.981249 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:19.971076 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:19.986898 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:19.989048 32608 raft_consensus.cc:1226] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:19.992178 1192 raft_consensus.cc:2354] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:19.931824 3477 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260806 04:01:20.011945 32602 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:19.995479 3704 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:19.994343 3590 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:19.996225 1183 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:01:19.995862 3274 consensus_queue.cc:237] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:19.998507 32614 raft_consensus.cc:2379] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:19.998773 3379 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:20.001075 32615 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:20.006227 32682 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:19.991521 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:20.053710 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 199986us
I20260806 04:01:20.051362 32427 leader_election.cc:398] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3.
I20260806 04:01:20.058796 3647 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.062769 32553 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:20.066960 32605 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:20.071676 32608 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:20.081367 32427 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:20.074760 3590 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:20.087657 32608 raft_consensus.cc:1226] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:20.084019 32321 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39754 (request call id 340) took 5478 ms. Trace:
I20260806 04:01:20.089493 1186 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
W20260806 04:01:20.092049 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 98091.2us
I20260806 04:01:20.098244 1136 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.101666 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:20.109253 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:20.112434 32322 catalog_manager.cc:5495] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:20.114488 32322 catalog_manager.cc:5495] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 2, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 2 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:01:20.115217 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:20.116478 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.116864 32609 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:01:20.118505 3724 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.118953 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.119745 32615 raft_consensus.cc:2354] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:20.112493 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:20.125700 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 44438.4us
I20260806 04:01:20.123343 32322 catalog_manager.cc:5495] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 4 to 5, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 5 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:20.091768 3437 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.088426 32321 rpcz_store.cc:276] 0806 04:01:14.605804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.608039 (+ 2235us) service_pool.cc:227] Handling call
0806 04:01:20.083955 (+5475916us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:19.092836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:19.093462 (+ 626us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:19.093511 (+ 49us) write_op.cc:435] Acquired schema lock
0806 04:01:19.093522 (+ 11us) tablet.cc:660] Decoding operations
0806 04:01:19.124524 (+ 31002us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:19.124619 (+ 95us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:19.124663 (+ 44us) tablet.cc:683] Acquiring locks for 3 operations
0806 04:01:19.125117 (+ 454us) tablet.cc:699] Row locks acquired
0806 04:01:19.125148 (+ 31us) write_op.cc:260] PREPARE: finished
0806 04:01:19.125324 (+ 176us) write_op.cc:270] Start()
0806 04:01:19.125517 (+ 193us) write_op.cc:276] Timestamp: P: 1785988879125252 usec, L: 0
0806 04:01:19.125563 (+ 46us) op_driver.cc:331] REPLICATION: starting
0806 04:01:19.129254 (+ 3691us) log.cc:842] Serialized 1316 byte log entry
0806 04:01:19.840760 (+711506us) op_driver.cc:444] REPLICATION: finished
0806 04:01:20.011352 (+170592us) write_op.cc:301] APPLY: starting
0806 04:01:20.011459 (+ 107us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:20.011618 (+ 159us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:20.011786 (+ 168us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:20.012438 (+ 652us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:20.012736 (+ 298us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 04:01:20.012807 (+ 71us) write_op.cc:312] APPLY: finished
0806 04:01:20.013155 (+ 348us) log.cc:842] Serialized 35 byte log entry
0806 04:01:20.013352 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:20.013456 (+ 104us) write_op.cc:454] Released schema lock
0806 04:01:20.013521 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:20.013801 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:20.013829 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":731845,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":170243,"apply.run_cpu_time_us":2802,"apply.run_wall_time_us":2802,"num_ops":3,"prepare.queue_time_us":165018,"prepare.run_cpu_time_us":7611,"prepare.run_wall_time_us":77765,"raft.queue_time_us":325084,"raft.run_cpu_time_us":13929,"raft.run_wall_time_us":117950,"replication_time_us":715095,"thread_start_us":392017,"threads_started":4,"wal-append.queue_time_us":115316}]]}
I20260806 04:01:20.128772 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:20.142688 32682 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:20.141228 32322 catalog_manager.cc:5495] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 1 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:20.149509 1189 raft_consensus.cc:2379] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:20.129652 3261 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:20.155071 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 52845.7us
I20260806 04:01:20.129434 3639 consensus_queue.cc:1046] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260806 04:01:20.161780 3639 raft_consensus.cc:3032] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:20.162169 3639 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:20.162987 3639 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.167306 3639 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:20.170162 32605 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:20.174361 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.164911 3637 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:20.195252 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 52498.1us
I20260806 04:01:20.197463 32427 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:20.199453 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.200261 3704 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.200897 3704 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:20.201059 3572 consensus_queue.cc:1046] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:20.203032 3547 raft_consensus.cc:3032] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:20.203406 3547 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:20.208279 3582 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:20.225937 32427 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.216528 3724 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.223867 3379 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260806 04:01:20.231552 3704 consensus_queue.cc:1046] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:01:20.231804 3721 consensus_queue.cc:1046] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260806 04:01:20.232470 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:20.233767 3172 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:20.233985 3724 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:20.234346 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:20.234560 3724 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:20.235224 3721 raft_consensus.cc:3032] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:20.235664 3721 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
W20260806 04:01:20.226997 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 104588us
I20260806 04:01:20.234357 3172 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:20.235260 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:20.239168 3172 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.239419 3569 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:20.236999 3218 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
W20260806 04:01:20.240198 3732 log.cc:924] Time spent T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260806 04:01:20.240018 3415 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:20.218155 3547 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.241161 3721 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:20.235881 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.242540 3379 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.234719 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:20.243585 32745 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:20.235615 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:20.225627 3367 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:20.246541 3367 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.248172 3697 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.248869 3367 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.249531 32466 raft_consensus.cc:1226] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:20.251614 3379 consensus_queue.cc:1046] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:20.252962 3379 raft_consensus.cc:3032] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:20.253345 3379 raft_consensus.cc:729] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:20.253446 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:20.254066 32470 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:20.254561 3379 consensus_queue.cc:260] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:20.255878 3379 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:20.256790 3704 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.257330 3582 raft_consensus.cc:3032] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:20.261096 3582 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:20.261358 32553 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.261581 32470 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:20.261693 3172 consensus_queue.cc:1046] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:20.262151 3531 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.262617 3724 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.242609 3547 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:20.263216 3134 raft_consensus.cc:3032] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:20.263626 3134 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:20.264416 3134 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:20.265578 3134 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:20.266799 32737 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:20.242486 3721 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:20.269408 32745 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:20.251667 3697 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:20.276355 32746 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:20.276961 32610 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:20.301076 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:20.302313 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.289863 32471 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:20.303479 32553 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:20.304360 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:20.304682 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:20.305902 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:20.303555 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:20.309048 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 87589.4us
I20260806 04:01:20.310333 1136 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:20.315543 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 89698.3us
I20260806 04:01:20.316192 32427 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.302029 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:20.323096 3200 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.324596 1136 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.312451 3172 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:20.326133 3172 raft_consensus.cc:686] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:20.326325 3367 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.309654 3292 consensus_queue.cc:1046] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.017s
I20260806 04:01:20.326968 3367 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:20.327472 3367 raft_consensus.cc:686] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:20.328416 3367 consensus_queue.cc:237] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.329162 3582 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.330545 3582 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:20.326150 32551 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.308218 3724 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:20.314392 3521 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.340786 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:20.340924 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.345227 3725 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.345580 3172 consensus_queue.cc:237] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.346033 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.346643 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.346997 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.347249 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:20.333053 3292 raft_consensus.cc:3032] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:20.347998 32614 raft_consensus.cc:1261] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:01:20.345407 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.366888 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:20.347671 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.363564 32553 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.360538 32603 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:20.367301 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:20.353057 32427 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.368018 1187 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:20.373064 3725 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.373147 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.373500 1185 raft_consensus.cc:1226] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:20.374136 32608 raft_consensus.cc:1226] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 04:01:20.375389 32608 raft_consensus.cc:1226] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 04:01:20.375892 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:20.376655 32682 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:20.318643 3531 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 04:01:20.375524 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:20.318962 3452 consensus_queue.cc:1046] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 04:01:20.385802 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.310777 3704 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260806 04:01:20.388202 32610 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:20.392596 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:20.394122 32427 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:20.395501 32427 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.396283 3697 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:20.396970 3704 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.397073 3367 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:20.397723 3367 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.399266 32682 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:20.399792 3367 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.400352 3367 raft_consensus.cc:3032] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:20.400692 3367 raft_consensus.cc:729] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:20.377149 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:20.402951 32427 leader_election.cc:398] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:20.406093 32427 leader_election.cc:401] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 04:01:20.412484 32610 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
W20260806 04:01:20.410681 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 69858.7us
I20260806 04:01:20.414824 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:20.382743 3415 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:20.411681 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:20.366969 3292 raft_consensus.cc:729] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:20.416358 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:20.419406 32746 raft_consensus.cc:2379] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:20.385190 3711 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.396705 3697 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:20.424598 1184 raft_consensus.cc:1226] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:20.424932 3637 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.406767 3261 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:20.433809 1190 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
W20260806 04:01:20.470306 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 100173us
I20260806 04:01:20.482807 32739 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:20.461926 1137 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:20.408849 3725 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:20.418146 3704 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
W20260806 04:01:20.493240 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 94069.1us
I20260806 04:01:20.412460 3521 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.430588 3647 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:20.507560 3776 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:20.518221 3776 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:20.495514 3261 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:20.428395 3569 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:20.520025 32683 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:20.518709 3776 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:20.434701 3697 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:20.428175 3283 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.091s
I20260806 04:01:20.588250 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:20.626159 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 106509us
I20260806 04:01:20.517051 3725 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.620200 3434 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:20.637462 3477 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:20.532984 3647 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.638470 3780 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:20.639012 3762 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:20.494948 3367 consensus_queue.cc:260] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.643767 1185 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:20.623377 3724 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:20.672894 3724 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:20.674799 3724 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.668746 3477 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.626608 3292 consensus_queue.cc:260] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.658044 3772 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:20.631379 3637 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:20.627126 3569 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.701668 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:20.702409 32473 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:20.704581 32553 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:20.676039 3639 consensus_meta.cc:287] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.508s user 0.003s sys 0.004s
I20260806 04:01:20.681354 3725 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:20.718261 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 289140us
I20260806 04:01:20.677317 3477 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.689030 3637 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:20.727689 32471 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:20.695878 3521 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:20.710223 3134 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:20.663570 3434 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:20.786738 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:20.792354 3647 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:20.793421 3434 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.793579 32612 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:20.735491 3134 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:20.794286 3134 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:20.683285 3775 consensus_queue.cc:1046] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20260806 04:01:20.730146 3637 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:20.816362 32473 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:20.820595 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:20.821488 32466 raft_consensus.cc:2379] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:20.671567 3780 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.831065 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:20.784888 3379 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:20.686323 3762 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.782104 3590 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
W20260806 04:01:20.832628 3582 consensus_meta.cc:287] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.502s user 0.010s sys 0.000s
I20260806 04:01:20.844277 3783 consensus_queue.cc:1046] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.120s
I20260806 04:01:20.847340 3742 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:20.800334 3521 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:20.858050 32471 raft_consensus.cc:2396] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260806 04:01:20.869274 32605 raft_consensus.cc:1261] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 04:01:20.864180 3547 consensus_meta.cc:287] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.599s user 0.008s sys 0.001s
I20260806 04:01:20.842736 3379 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.863394 3782 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 04:01:20.885648 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 365728us
I20260806 04:01:20.902408 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:20.903352 32746 raft_consensus.cc:2379] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:20.918288 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:20.857192 3742 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.921934 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:20.901288 3782 raft_consensus.cc:686] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:20.947465 32746 raft_consensus.cc:1261] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:20.956641 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:20.956630 3759 log.cc:924] Time spent T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20260806 04:01:20.960338 3794 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:20.961040 3794 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.963194 3794 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:20.968518 3762 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.972590 3697 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:20.974733 3697 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:20.925253 3569 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.993444 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.001025 32466 raft_consensus.cc:2379] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:20.987950 32602 raft_consensus.cc:2396] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:01:21.008002 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.006388 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.009366 1189 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:21.013536 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.015208 1192 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:21.015830 32551 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.035883 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:21.034430 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 399415us
I20260806 04:01:21.038169 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.044900 32466 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:01:21.012977 3801 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260806 04:01:21.048900 32749 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 04:01:21.049855 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 231165us
I20260806 04:01:21.055922 32427 leader_election.cc:398] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:21.056972 32427 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.038976 3721 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:21.058410 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.063396 32427 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.014827 3569 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260806 04:01:21.082957 32682 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:21.053402 3521 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:20.999065 3776 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.100332 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.065416 3721 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:21.047703 3531 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260806 04:01:21.061331 3582 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:21.092540 3379 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.092455 3780 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:21.106905 32745 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:21.108744 32477 raft_consensus.cc:1226] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:21.115283 3134 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:21.104519 1183 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:21.124511 3810 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:21.123706 3744 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:21.116611 3766 consensus_queue.cc:1035] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260806 04:01:21.064253 3647 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20260806 04:01:21.115550 3784 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260806 04:01:21.144670 32746 raft_consensus.cc:1226] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:21.116250 3798 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:21.150173 32553 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:21.054158 3797 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260806 04:01:21.103677 3794 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.174993 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.148561 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.113427 3721 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:21.192636 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:21.196760 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 113902us
W20260806 04:01:21.197500 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 97040.7us
I20260806 04:01:21.207152 3704 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:21.196110 3261 consensus_meta.cc:287] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.607s user 0.011s sys 0.000s
W20260806 04:01:21.208623 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 172657us
I20260806 04:01:21.053112 3782 consensus_queue.cc:237] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:21.207793 32682 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:21.217108 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.218822 32614 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:01:21.199600 3798 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:21.197988 3569 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:21.195520 3742 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.247473 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.212575 3531 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:21.248209 32603 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:21.194180 3794 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.242748 3261 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:21.247041 3569 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.257725 3845 consensus_queue.cc:1046] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:21.246233 3798 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.250841 3531 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:21.207775 3704 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.244808 3283 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.264669 3776 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.279738 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:21.171586 3744 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.281459 3704 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.283144 3283 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.283983 3134 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.285334 3582 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:21.267848 3790 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:21.285635 3569 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.287067 3798 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:21.291239 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 83561.7us
I20260806 04:01:21.285800 3531 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:21.309067 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 135247us
I20260806 04:01:21.311502 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:21.319612 32473 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
W20260806 04:01:21.320794 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 225005us
I20260806 04:01:21.325901 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.295228 3810 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 04:01:21.298069 3826 log.cc:924] Time spent T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260806 04:01:21.328706 3647 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:21.333084 3822 log.cc:924] Time spent T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20260806 04:01:21.333748 3829 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:21.334494 3274 raft_consensus.cc:3032] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:21.336369 3782 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:21.345324 3521 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.337567 3794 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.339326 32682 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:21.350844 3283 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.350093 3782 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.355165 3782 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.355927 3818 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:21.356525 3818 raft_consensus.cc:686] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:21.356781 3766 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:21.357448 3818 consensus_queue.cc:237] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.359127 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.359712 32478 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:21.360519 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.361059 32478 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:21.361423 3839 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:21.362164 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.342839 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.333758 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.337482 3851 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:21.346961 3647 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:21.363961 32473 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:21.343600 3569 consensus_queue.cc:1035] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260806 04:01:21.351065 32603 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:21.371876 32475 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:21.348789 3274 raft_consensus.cc:729] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:21.368068 3851 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.368585 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.350150 3521 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.377496 32604 raft_consensus.cc:2362] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4. Re-sending same reply.
I20260806 04:01:21.374928 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:21.368892 3829 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.300812 3637 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:21.345870 3810 raft_consensus.cc:686] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:21.377985 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1778) took 1042 ms. Trace:
I20260806 04:01:21.377924 32683 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:21.388128 32604 rpcz_store.cc:276] 0806 04:01:20.335759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.336144 (+ 385us) service_pool.cc:227] Handling call
0806 04:01:21.368438 (+1032294us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b58001e5798. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:21.377932 (+ 9494us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12947840}
I20260806 04:01:21.381460 32608 raft_consensus.cc:1226] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 04:01:21.388643 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:21.388828 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:21.389567 32745 raft_consensus.cc:2379] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:21.382009 32615 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:21.370141 3647 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:21.383703 1137 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:21.390822 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.339300 3783 consensus_queue.cc:1046] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260806 04:01:21.394992 32478 raft_consensus.cc:2379] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:21.388473 3801 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:21.397650 3810 consensus_queue.cc:237] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.398380 3790 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.398432 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.399084 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.398105 3274 consensus_queue.cc:260] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.398073 3801 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:21.400470 32748 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:21.397604 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.401683 32745 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:21.401760 32604 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:21.407029 32682 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:21.394109 3782 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:21.412048 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.396158 3452 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:21.412757 32745 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:21.415769 3842 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:21.412164 3782 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:21.416105 3647 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.417253 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.417716 3721 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.421904 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.422068 3867 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:21.422226 3829 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.423517 3521 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.424863 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:21.427273 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.408795 3801 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:21.428247 1183 raft_consensus.cc:1226] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 04:01:21.444998 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.429729 3637 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:21.431928 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33964 (request call id 788) took 1118 ms. Trace:
I20260806 04:01:21.432165 3531 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.433571 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:21.446967 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.434464 32428 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:21.447607 1188 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:21.440328 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:21.448251 32608 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:21.445652 32745 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:21.432713 3776 consensus_queue.cc:1046] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:21.450680 3531 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.452455 3582 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:21.452896 3582 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:21.444702 3274 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:21.451880 3721 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.454535 3798 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260806 04:01:21.449342 3865 log.cc:924] Time spent T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.052s user 0.002s sys 0.000s
I20260806 04:01:21.435026 3842 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:21.445791 32743 rpcz_store.cc:276] 0806 04:01:20.313757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.313882 (+ 125us) service_pool.cc:227] Handling call
0806 04:01:21.430984 (+1117102us) spinlock_profiling.cc:243] Waited 9.61 ms on lock 0x7b5800004298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e6726 00007fd1721e6900 00007fd172199b50 00007fd16b890807 00007fd16b890792 00007fd16b8906c1
0806 04:01:21.431859 (+ 875us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22105344}
I20260806 04:01:21.432291 3693 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:21.438035 3782 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:21.461692 32682 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:21.438618 3867 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:21.453301 3582 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:21.433945 3784 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260806 04:01:21.410892 3790 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.445178 3303 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.498837 32682 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:21.504392 3840 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.505249 3840 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.474969 3842 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:21.440995 3647 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.440783 3452 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.520838 3797 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260806 04:01:21.524411 1188 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:21.525792 3790 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.526705 3521 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.527657 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:21.527515 3521 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.529567 1186 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:21.554095 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.552100 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:21.538409 3521 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.512297 3261 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:21.584440 3261 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.513296 3693 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.474483 3867 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:21.526742 3779 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:21.588435 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.589077 1186 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
W20260806 04:01:21.589258 3740 log.cc:924] Time spent T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 1.222s user 0.000s sys 0.003s
I20260806 04:01:21.593137 3647 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:21.539875 3768 log.cc:924] Time spent T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.951s user 0.000s sys 0.020s
I20260806 04:01:21.566797 3704 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:21.603195 3704 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:21.580314 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.587742 3693 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.582006 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.607501 1189 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
W20260806 04:01:21.516351 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 190534us
I20260806 04:01:21.613818 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:21.591966 3261 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.615157 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.615159 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.581488 3840 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.616412 1188 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:21.617362 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:21.617908 1188 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:21.553136 3829 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:21.558346 32612 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1777) took 1222 ms (client timeout 1536 ms). Trace:
I20260806 04:01:21.619174 3829 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.623355 32608 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
W20260806 04:01:21.619537 32612 rpcz_store.cc:269] 0806 04:01:20.335468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.339713 (+ 4245us) service_pool.cc:227] Handling call
0806 04:01:21.558105 (+1218392us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54003340d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:21.558265 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":850834176}
I20260806 04:01:21.625655 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.521286 3303 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.551443 3851 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.606652 3697 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:21.657984 3697 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.660777 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.656595 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1653) took 1431 ms. Trace:
I20260806 04:01:21.661818 32739 rpcz_store.cc:276] 0806 04:01:20.224838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.225189 (+ 351us) service_pool.cc:227] Handling call
0806 04:01:20.242836 (+ 17647us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:20.484278 (+241442us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:21.656338 (+1172060us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54003dfed4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:21.656524 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":551447424}
I20260806 04:01:21.550725 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1775) took 1215 ms. Trace:
I20260806 04:01:21.671672 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.673022 32749 raft_consensus.cc:2379] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:21.673834 1136 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.683358 3704 leader_election.cc:288] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.684290 3582 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.703527 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.704497 32475 raft_consensus.cc:2354] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:21.661031 3829 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.703774 3582 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:21.708242 3737 log.cc:924] Time spent T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 1.369s user 0.003s sys 0.000s
I20260806 04:01:21.704942 1188 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:21.710987 1188 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 04:01:21.711748 3697 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:21.714516 3743 log.cc:924] Time spent T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 1.364s user 0.000s sys 0.002s
I20260806 04:01:21.685153 32743 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260806 04:01:21.684842 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.686657 32608 raft_consensus.cc:2379] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:21.718001 32743 pending_rounds.cc:76] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Aborting all ops after (but not including) 4
I20260806 04:01:21.718358 32743 pending_rounds.cc:97] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260806 04:01:21.718609 32743 raft_consensus.cc:2866] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:21.718815 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.719465 32608 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:21.719779 1186 raft_consensus.cc:2379] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:21.720279 3303 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.720584 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.683386 32602 rpcz_store.cc:276] 0806 04:01:20.334831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.339674 (+ 4843us) service_pool.cc:227] Handling call
0806 04:01:20.830954 (+491280us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b58001e5798. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4c6c 00007fd16a2f4bd6 000000000034aea9 00007fd172120a30 00007fd1721e67b4 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:21.550401 (+719447us) spinlock_profiling.cc:243] Waited 83.5 ms on lock 0x7b54003a1bd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:21.550634 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":221712384}
I20260806 04:01:21.722837 32608 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 04:01:21.723372 32322 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39782 (request call id 332) took 6991 ms. Trace:
I20260806 04:01:21.697863 3756 raft_consensus.cc:3032] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:21.695399 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:21.724419 3756 raft_consensus.cc:729] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:21.725629 3756 consensus_queue.cc:260] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.727522 3756 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:21.698187 3801 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.700721 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.723948 32608 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 04:01:21.550668 3452 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:21.738624 32470 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:21.741245 32553 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:21.548476 3853 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:21.744925 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 83641.1us
I20260806 04:01:21.745404 3829 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:21.746183 3753 log.cc:924] Time spent T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 1.225s user 0.002s sys 0.004s
I20260806 04:01:21.747051 3845 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:21.745044 32427 leader_election.cc:302] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:21.752574 1190 raft_consensus.cc:2354] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5.
W20260806 04:01:21.769181 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 95447us
I20260806 04:01:21.770032 1136 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.773524 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.771915 1136 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:21.723711 32322 rpcz_store.cc:276] 0806 04:01:14.732462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:14.733839 (+ 1377us) service_pool.cc:227] Handling call
0806 04:01:21.723316 (+6989477us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:20.411724 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:20.426902 (+ 15178us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:20.426942 (+ 40us) write_op.cc:435] Acquired schema lock
0806 04:01:20.426959 (+ 17us) tablet.cc:660] Decoding operations
0806 04:01:20.427600 (+ 641us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:20.427662 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:20.427688 (+ 26us) tablet.cc:683] Acquiring locks for 4 operations
0806 04:01:20.428052 (+ 364us) tablet.cc:699] Row locks acquired
0806 04:01:20.428070 (+ 18us) write_op.cc:260] PREPARE: finished
0806 04:01:20.494359 (+ 66289us) write_op.cc:270] Start()
0806 04:01:20.519951 (+ 25592us) write_op.cc:276] Timestamp: P: 1785988880484310 usec, L: 0
0806 04:01:20.523284 (+ 3333us) op_driver.cc:331] REPLICATION: starting
0806 04:01:20.683433 (+160149us) log.cc:842] Serialized 1708 byte log entry
0806 04:01:21.104682 (+421249us) op_driver.cc:444] REPLICATION: finished
0806 04:01:21.358246 (+253564us) write_op.cc:301] APPLY: starting
0806 04:01:21.358326 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:21.358473 (+ 147us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:21.358597 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:21.359220 (+ 623us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:21.359400 (+ 180us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0806 04:01:21.359440 (+ 40us) write_op.cc:312] APPLY: finished
0806 04:01:21.359647 (+ 207us) log.cc:842] Serialized 41 byte log entry
0806 04:01:21.359791 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:21.359898 (+ 107us) write_op.cc:454] Released schema lock
0806 04:01:21.359933 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:21.360181 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:21.360215 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":64990,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":245495,"apply.run_cpu_time_us":2229,"apply.run_wall_time_us":2229,"num_ops":4,"prepare.queue_time_us":220280,"prepare.run_cpu_time_us":15259,"prepare.run_wall_time_us":298471,"replication_time_us":570513,"thread_start_us":210545,"threads_started":2}]]}
I20260806 04:01:21.781221 3704 consensus_queue.cc:1046] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:21.781540 3694 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:21.782397 3704 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:21.771591 3829 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:21.782903 3704 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:21.787346 3449 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:21.765437 3867 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.695917 3782 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.792793 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 392) took 1784 ms. Trace:
I20260806 04:01:21.794471 3292 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:21.795998 1187 raft_consensus.cc:2354] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:21.776271 3200 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:21.773423 3218 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:21.797791 3200 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:21.805274 3452 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:21.784886 3704 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:21.806439 3867 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.752856 3853 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.809906 3782 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:21.798087 3829 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.810597 3744 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.813529 3801 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:21.773635 3845 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.776510 3780 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:21.817234 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:21.841722 32741 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 6
I20260806 04:01:21.819454 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.820520 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:21.843786 32746 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:01:21.844374 32427 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:21.831116 3724 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:21.829629 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:21.831336 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.835503 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:21.849149 32466 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:21.824506 3200 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:21.806552 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:21.826857 3452 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:21.801976 3842 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:21.846413 32551 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:21.782357 3694 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:21.848587 32477 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:21.878508 32551 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:21.847653 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 348882us
I20260806 04:01:21.850332 32427 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:21.843838 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.865757 32741 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
W20260806 04:01:21.827879 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 86713.1us
I20260806 04:01:21.860872 1137 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.891640 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:21.895480 32682 leader_election.cc:302] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:21.899938 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.846532 3704 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:21.895663 1137 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:21.902679 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.904654 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:21.900599 1186 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:21.855238 3790 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:21.918450 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:21.876331 3452 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:21.921197 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:21.922364 32682 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:21.920634 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:21.923630 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:21.924153 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:21.926407 32471 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:01:21.894316 3851 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:21.890455 3797 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:21.847510 3840 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260806 04:01:21.933542 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.934193 32608 raft_consensus.cc:2379] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:21.888383 3547 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:21.878135 3724 raft_consensus.cc:686] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:21.932734 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:21.855571 3636 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
W20260806 04:01:21.945395 32614 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.014s sys 0.019s
I20260806 04:01:21.933413 1136 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:21.904134 3818 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:21.902107 3569 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:21.974048 32608 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:21.938694 3274 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:21.951647 3793 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:21.953908 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:21.985265 1189 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 6
I20260806 04:01:21.925657 3697 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:21.961524 3134 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:21.960906 32326 catalog_manager.cc:5495] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:21.956158 3772 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 04:01:21.952263 3725 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.023s sys 0.023s
I20260806 04:01:21.930449 3839 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:21.966604 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:21.968917 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:21.969537 32614 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.014s sys 0.020s
W20260806 04:01:22.039564 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 106898us
I20260806 04:01:21.970688 32605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1774) took 1625 ms. Trace:
I20260806 04:01:22.040377 32605 rpcz_store.cc:276] 0806 04:01:20.334518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.343458 (+ 8940us) service_pool.cc:227] Handling call
0806 04:01:20.716913 (+373455us) spinlock_profiling.cc:243] Waited 8.86 ms on lock 0x7b58001e5798. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:20.717130 (+ 217us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:20.869660 (+152530us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:21.959623 (+1089963us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540007dfd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:21.959802 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":365969920}
I20260806 04:01:21.950795 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:22.042279 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:21.981999 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 86398.4us
I20260806 04:01:21.944165 3851 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:21.982151 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:21.993638 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 60318.8us
I20260806 04:01:21.793478 32738 rpcz_store.cc:276] 0806 04:01:20.008658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.008828 (+ 170us) service_pool.cc:227] Handling call
0806 04:01:20.009195 (+ 367us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:20.009641 (+ 446us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:20.009676 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:20.010170 (+ 494us) log.cc:842] Serialized 26 byte log entry
0806 04:01:20.376109 (+365939us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:20.376167 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:20.376244 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:21.745240 (+1368996us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b2400181e98. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a184cf4 00007fd16a182dfb 00007fd16a211559 00007fd16a2111e5 00007fd16a21082d 00007fd16cebf46e 00007fd16ced6c0c 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551
0806 04:01:21.790330 (+ 45090us) raft_consensus.cc:1671] finished
0806 04:01:21.790353 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:21.792692 (+ 2339us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1286,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"spinlock_wait_cycles":3434752,"thread_start_us":341437,"threads_started":1,"wal-append.queue_time_us":392002}
I20260806 04:01:22.058815 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:22.060122 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:22.060750 32473 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:22.026399 3134 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:22.064852 3724 consensus_queue.cc:237] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:22.060699 3873 log.cc:924] Time spent T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20260806 04:01:22.025097 3725 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.942s user 0.023s sys 0.024s
I20260806 04:01:21.990497 3697 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:22.034674 3836 raft_consensus.cc:2726] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:22.047019 32748 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:22.046766 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 48097.1us
I20260806 04:01:22.054616 3814 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:22.051604 32427 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:22.053926 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:22.025979 3379 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:22.060746 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:22.042894 1186 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:21.996901 32326 catalog_manager.cc:5495] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 2 to 3, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 3 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:22.070250 32471 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:21.998616 3818 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:21.967784 3547 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:22.026809 3793 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:22.097007 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:22.038884 3766 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:22.086859 32741 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:22.114012 32326 catalog_manager.cc:5495] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260806 04:01:22.120209 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 59889.7us
I20260806 04:01:22.056408 3851 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:22.122342 1186 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:22.039093 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:22.127668 32326 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 6 to 7, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 7 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:22.124763 32746 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:22.131884 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:22.125985 32683 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:22.091536 3697 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:22.132491 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:22.067190 3134 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:22.147492 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:22.056334 3647 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:22.148669 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:22.107434 3840 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:22.119557 3274 raft_consensus.cc:686] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:22.154486 32746 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:22.110844 3793 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:22.152281 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:22.180003 3882 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:22.183555 32469 raft_consensus.cc:1261] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:22.151623 3174 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:22.221259 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:22.142745 3853 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:22.107195 3815 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.323s
I20260806 04:01:22.146019 3790 consensus_queue.cc:1046] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260806 04:01:22.135206 3751 consensus_queue.cc:1035] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20260806 04:01:22.143282 1184 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:01:22.234802 32478 raft_consensus.cc:2379] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:22.162086 3829 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:22.163791 3814 raft_consensus.cc:686] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:22.172294 3803 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:22.227748 3882 raft_consensus.cc:686] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:22.270110 1187 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 4 LEARNER]: Advancing to term 6
I20260806 04:01:22.229472 3274 consensus_queue.cc:237] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:22.069305 3839 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:22.203902 3802 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:22.275547 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:22.272456 32746 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:22.224570 3842 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:22.147397 3870 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.376s
I20260806 04:01:22.183084 3647 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:22.298049 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:22.281358 32477 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:22.305389 1187 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 6 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:22.325177 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:22.321655 3893 log.cc:924] Time spent T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
I20260806 04:01:22.323865 3803 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:22.334033 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1661) took 1824 ms. Trace:
I20260806 04:01:22.259681 3694 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:22.336292 32737 rpcz_store.cc:276] 0806 04:01:20.509727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.519508 (+ 9781us) service_pool.cc:227] Handling call
0806 04:01:20.633655 (+114147us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:20.707905 (+ 74250us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:20.714607 (+ 6702us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:21.783591 (+1068984us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b2c00108868. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:21.784220 (+ 629us) log.cc:842] Serialized 26 byte log entry
0806 04:01:21.993026 (+208806us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:21.998456 (+ 5430us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:22.333638 (+335182us) spinlock_profiling.cc:243] Waited 54.9 ms on lock 0x7b54003a5cd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:22.333694 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:22.333765 (+ 71us) raft_consensus.cc:1671] finished
0806 04:01:22.333783 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:22.333954 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3629,"raft.queue_time_us":30200,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6062,"spinlock_wait_cycles":136737280,"thread_start_us":185765,"threads_started":1,"wal-append.queue_time_us":446269}
I20260806 04:01:22.341584 3803 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:22.345219 3547 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:22.287096 3884 log.cc:924] Time spent T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20260806 04:01:22.350930 32745 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:22.295002 3174 raft_consensus.cc:686] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:22.421216 3809 log.cc:924] Time spent T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 1.201s user 0.000s sys 0.006s
I20260806 04:01:22.444502 32748 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
W20260806 04:01:22.305171 3894 log.cc:924] Time spent T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.076s user 0.000s sys 0.002s
I20260806 04:01:22.310901 3367 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:22.258409 3779 consensus_queue.cc:1035] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.113s
I20260806 04:01:22.421769 3174 consensus_queue.cc:237] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:22.473038 3704 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.554s user 0.007s sys 0.006s
I20260806 04:01:22.298607 3802 raft_consensus.cc:686] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:22.289757 3845 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:22.525244 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:22.560791 3889 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:22.561870 3889 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:22.571277 3802 consensus_queue.cc:237] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:22.574900 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 521) took 2194 ms. Trace:
I20260806 04:01:22.446280 3800 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:22.575527 32609 rpcz_store.cc:276] 0806 04:01:20.380501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.418559 (+ 38058us) service_pool.cc:227] Handling call
0806 04:01:20.434768 (+ 16209us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:20.484501 (+ 49733us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:20.484530 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:20.519562 (+ 35032us) log.cc:842] Serialized 26 byte log entry
0806 04:01:20.928426 (+408864us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:21.170805 (+242379us) log.cc:842] Serialized 14 byte log entry
0806 04:01:21.171004 (+ 199us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:21.171088 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:22.126120 (+955032us) spinlock_profiling.cc:243] Waited 5.54 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:22.566564 (+440444us) raft_consensus.cc:1671] finished
0806 04:01:22.574624 (+ 8060us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:22.574784 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":274973,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":35897,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":17741,"spinlock_wait_cycles":12731776,"thread_start_us":437716,"threads_started":2,"wal-append.queue_time_us":418400}
I20260806 04:01:22.302251 3829 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:22.277606 3818 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:22.488890 3839 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:22.574864 3882 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:22.576719 3800 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:22.558522 3814 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:22.985409 32466 consensus_meta.cc:287] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 1.123s user 0.007s sys 0.007s
I20260806 04:01:22.985872 32466 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:22.986049 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 407) took 1577 ms. Trace:
W20260806 04:01:22.716650 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.032s sys 0.071s
W20260806 04:01:22.987646 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.868s user 0.034s sys 0.076s
I20260806 04:01:22.986475 32739 rpcz_store.cc:276] 0806 04:01:21.408901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.668938 (+260037us) service_pool.cc:227] Handling call
0806 04:01:21.669314 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:21.669729 (+ 415us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:21.669757 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:21.670237 (+ 480us) log.cc:842] Serialized 26 byte log entry
0806 04:01:21.949173 (+278936us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:21.949241 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:21.949346 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:22.985838 (+1036492us) raft_consensus.cc:1671] finished
0806 04:01:22.985861 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:22.985992 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":952,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":131,"thread_start_us":265489,"threads_started":1,"wal-append.queue_time_us":496779}
W20260806 04:01:22.822108 1189 consensus_meta.cc:287] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.836s user 0.006s sys 0.005s
W20260806 04:01:22.938153 1139 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.049s sys 0.081s
I20260806 04:01:22.993903 1189 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:22.999094 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55048 (request call id 646) took 1169 ms. Trace:
W20260806 04:01:22.999658 1139 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.961s user 0.055s sys 0.082s
I20260806 04:01:22.998308 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 781) took 1138 ms. Trace:
I20260806 04:01:23.001116 1189 rpcz_store.cc:276] 0806 04:01:21.859490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.931457 (+ 71967us) service_pool.cc:227] Handling call
0806 04:01:22.998249 (+1066792us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:22.979707 3793 consensus_meta.cc:287] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.753s user 0.003s sys 0.008s
W20260806 04:01:22.976783 3200 consensus_meta.cc:287] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 1.088s user 0.007s sys 0.007s
W20260806 04:01:22.852319 32682 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 1.798s (SENT) [suppressed 8 similar messages]
W20260806 04:01:22.785667 3809 log.cc:924] Time spent T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20260806 04:01:23.026522 3704 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:22.950065 3647 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.000190 3889 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:22.999514 32466 rpcz_store.cc:276] 0806 04:01:21.817039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.817179 (+ 140us) service_pool.cc:227] Handling call
0806 04:01:22.986351 (+1169172us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:22.851284 3878 log.cc:924] Time spent T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.440s user 0.001s sys 0.000s
W20260806 04:01:22.998502 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.023s sys 0.065s
I20260806 04:01:23.029174 32603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 1986) took 1652 ms. Trace:
W20260806 04:01:23.029364 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.891s user 0.023s sys 0.065s
I20260806 04:01:23.029631 32603 rpcz_store.cc:276] 0806 04:01:21.376888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.381445 (+ 4557us) service_pool.cc:227] Handling call
0806 04:01:21.381770 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:21.391867 (+ 10097us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:21.391898 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:21.396562 (+ 4664us) log.cc:842] Serialized 26 byte log entry
0806 04:01:21.843207 (+446645us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:21.843245 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:21.843307 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:23.028956 (+1185649us) raft_consensus.cc:1671] finished
0806 04:01:23.028975 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:23.029110 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2570,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":4197,"spinlock_wait_cycles":25600,"thread_start_us":398825,"threads_started":1,"wal-append.queue_time_us":531422}
I20260806 04:01:23.030994 3797 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:23.032426 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 900458us
I20260806 04:01:23.035885 32749 raft_consensus.cc:1226] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
W20260806 04:01:22.963200 3452 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.018s user 0.006s sys 0.005s
W20260806 04:01:22.870107 32740 consensus_meta.cc:287] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.908s user 0.011s sys 0.002s
W20260806 04:01:22.943675 3756 consensus_meta.cc:287] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 1.216s user 0.010s sys 0.006s
I20260806 04:01:23.031859 3797 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:22.942162 3379 consensus_meta.cc:287] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.840s user 0.007s sys 0.003s
I20260806 04:01:22.858285 3829 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:22.815900 3639 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:23.052248 3829 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.052492 3639 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.053030 3829 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.056895 3639 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:23.057907 3639 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:22.849648 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 1991) took 1269 ms. Trace:
I20260806 04:01:23.058269 3797 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:23.058758 32612 rpcz_store.cc:276] 0806 04:01:21.580725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.728894 (+148169us) service_pool.cc:227] Handling call
0806 04:01:21.729271 (+ 377us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:21.775232 (+ 45961us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:21.777433 (+ 2201us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:21.859517 (+ 82084us) log.cc:842] Serialized 26 byte log entry
0806 04:01:22.161151 (+301634us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:22.170239 (+ 9088us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:22.200903 (+ 30664us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:22.849268 (+648365us) raft_consensus.cc:1671] finished
0806 04:01:22.849297 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:22.849547 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12500,"raft.queue_time_us":38346,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":13388,"thread_start_us":160712,"threads_started":1,"wal-append.queue_time_us":913619}
I20260806 04:01:22.855441 3779 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:22.797191 3751 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:23.039760 3200 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:23.069928 3200 leader_election.cc:288] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:23.027546 3704 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.072862 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.075273 3704 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:23.075706 1183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 774) took 1541 ms (client timeout 1773 ms). Trace:
W20260806 04:01:23.076320 1183 rpcz_store.cc:269] 0806 04:01:21.534619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.575145 (+ 40526us) service_pool.cc:227] Handling call
0806 04:01:23.072752 (+1497607us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4c6c 00007fd16a2f4bd6 000000000034aea9 00007fd172120a30 00007fd1721e67b4 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:23.075636 (+ 2884us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25375488}
I20260806 04:01:23.067498 3379 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:23.082072 32605 raft_consensus.cc:1261] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:23.082398 3693 raft_consensus.cc:3032] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:23.082952 3693 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:23.083463 32605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1806) took 1144 ms. Trace:
I20260806 04:01:23.083907 32605 rpcz_store.cc:276] 0806 04:01:21.939128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:22.098651 (+159523us) service_pool.cc:227] Handling call
0806 04:01:22.137147 (+ 38496us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:23.082423 (+945276us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:23.083211 (+ 788us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b540034e9d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:23.083396 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2172205952}
I20260806 04:01:23.036726 3793 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.086061 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1802) took 1164 ms. Trace:
I20260806 04:01:23.063117 3779 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:23.083508 3697 consensus_meta.cc:287] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.921s user 0.009s sys 0.002s
I20260806 04:01:23.092433 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1799) took 2055 ms. Trace:
I20260806 04:01:23.101878 32474 rpcz_store.cc:276] 0806 04:01:21.037395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.037607 (+ 212us) service_pool.cc:227] Handling call
0806 04:01:23.091681 (+2054074us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b5800003398. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:23.091899 (+ 218us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:23.092097 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:23.092123 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:23.092164 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:23.092194 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:23.092258 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:23.092374 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5770112}
I20260806 04:01:23.048210 32740 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:23.089596 3800 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.048027 3452 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.100782 3855 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:23.110499 3704 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:23.112108 3704 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:23.121924 32683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.025s sys 0.025s
W20260806 04:01:23.124119 32683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.883s user 0.025s sys 0.026s
I20260806 04:01:23.086663 32602 rpcz_store.cc:276] 0806 04:01:21.921474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.923884 (+ 2410us) service_pool.cc:227] Handling call
0806 04:01:21.962500 (+ 38616us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:22.106907 (+144407us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:22.118989 (+ 12082us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:22.125197 (+ 6208us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:22.140149 (+ 14952us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:23.084560 (+944411us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b5400353754. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:23.084616 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:23.085970 (+ 1354us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":242225920}
I20260806 04:01:23.089411 3693 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:23.136181 32326 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.893s user 0.012s sys 0.017s
I20260806 04:01:23.136884 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:23.137141 32551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.018s sys 0.007s
W20260806 04:01:23.138748 32551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.019s user 0.018s sys 0.007s
W20260806 04:01:23.121587 3725 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.008s sys 0.011s
I20260806 04:01:23.137594 32474 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:23.100741 3697 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.148231 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.148926 32466 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:23.087484 3639 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:23.140986 3725 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.998s user 0.008s sys 0.011s
I20260806 04:01:23.156486 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.158198 3704 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:23.162956 32609 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:23.160750 3531 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:23.142064 3693 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:23.165848 32467 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:23.138805 3829 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:23.139310 3521 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.969s user 0.007s sys 0.024s
I20260806 04:01:23.111742 3855 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:23.136590 32326 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.904s user 0.012s sys 0.017s
I20260806 04:01:23.168924 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:23.138622 3283 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.009s sys 0.016s
I20260806 04:01:23.178368 3639 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:23.179311 3851 consensus_meta.cc:287] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.036s user 0.012s sys 0.001s
I20260806 04:01:23.180147 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 403) took 1818 ms. Trace:
W20260806 04:01:23.178174 32740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2123) took 2902 ms (client timeout 1727 ms). Trace:
I20260806 04:01:23.181396 3855 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:23.181481 32743 rpcz_store.cc:276] 0806 04:01:21.361925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.464297 (+102372us) service_pool.cc:227] Handling call
0806 04:01:21.492296 (+ 27999us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:21.719046 (+226750us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:23.179906 (+1460860us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54000fb754. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:23.180068 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":442594560}
I20260806 04:01:23.184281 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 772) took 1667 ms. Trace:
I20260806 04:01:23.088238 3751 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:23.171357 3521 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.002s user 0.007s sys 0.025s
I20260806 04:01:23.184723 1192 rpcz_store.cc:276] 0806 04:01:21.516373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.516500 (+ 127us) service_pool.cc:227] Handling call
0806 04:01:21.528676 (+ 12176us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:21.558195 (+ 29519us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:21.558221 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:21.558714 (+ 493us) log.cc:842] Serialized 26 byte log entry
0806 04:01:21.879504 (+320790us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:21.879548 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:23.183764 (+1304216us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b540062edd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:23.183807 (+ 43us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:23.183868 (+ 61us) raft_consensus.cc:1671] finished
0806 04:01:23.183885 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:23.184199 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2639,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":20968,"spinlock_wait_cycles":66950016,"thread_start_us":288326,"threads_started":1,"wal-append.queue_time_us":477324,"wal-append.run_cpu_time_us":2028,"wal-append.run_wall_time_us":1007039}
W20260806 04:01:23.184814 3367 consensus_meta.cc:287] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.665s user 0.004s sys 0.008s
I20260806 04:01:23.207904 32683 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:23.180418 3903 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.212607 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.224781 32478 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:23.224996 3772 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:01:23.180893 32740 rpcz_store.cc:269] 0806 04:01:20.275297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:20.308125 (+ 32828us) service_pool.cc:227] Handling call
0806 04:01:21.527495 (+1219370us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b5800004298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:23.177893 (+1650398us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b540011d5d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:23.178083 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1031496192}
I20260806 04:01:23.225507 3772 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:23.225978 3772 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:23.191946 3283 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s user 0.009s sys 0.016s
I20260806 04:01:23.221354 32605 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:23.221174 3521 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:23.221208 32683 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:23.235276 3756 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:23.235819 3756 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:23.236500 3756 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:23.213330 3725 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:23.229614 1186 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:23.202051 3793 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.186236 3829 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:23.179848 3851 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.242199 3725 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.246138 32683 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:23.254086 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.254781 32609 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:23.255852 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55048 (request call id 653) took 1210 ms. Trace:
I20260806 04:01:23.256390 32467 rpcz_store.cc:276] 0806 04:01:22.045192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:22.045304 (+ 112us) service_pool.cc:227] Handling call
0806 04:01:23.255679 (+1210375us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001b1054. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:23.255786 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2479890816}
I20260806 04:01:23.185137 3531 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.261269 32427 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:23.243197 3829 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:23.270254 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.270548 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.283769 32608 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:23.282629 32605 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:23.243896 3854 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:23.292354 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.241369 3521 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:23.224784 3903 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.334375 3452 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:23.294490 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:23.392664 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 110203us
I20260806 04:01:23.292564 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.397933 32467 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:23.335697 3636 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.399657 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.275609 3794 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.403227 3794 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.404363 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.327224 3531 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.405092 1192 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:23.406257 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.407079 1183 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:23.408419 3751 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.325146 3697 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.409029 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.412133 32427 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:23.412640 3794 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.413733 3521 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:23.275696 3647 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:23.423830 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.413460 3811 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:23.424163 3647 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:23.425412 3647 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:23.426950 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.428675 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:23.429344 32609 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:23.420468 3903 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.422842 3531 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.432458 3531 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.335904 3779 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:23.418718 3780 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.117s user 0.014s sys 0.044s
I20260806 04:01:23.399708 3636 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.442273 32683 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:23.444629 3531 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.446934 3851 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.280323 3283 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:23.455287 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:23.459276 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.484407 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.454118 3794 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:23.460822 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.510679 1189 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
W20260806 04:01:23.438329 3780 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.149s user 0.014s sys 0.044s
I20260806 04:01:23.472503 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.290915 3891 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.486408 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:23.494560 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.494910 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:41296 (request call id 1090) took 2012 ms. Trace:
I20260806 04:01:23.504150 3829 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:23.460934 1136 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:23.505545 3725 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:23.513751 32475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55048 (request call id 640) took 1922 ms (client timeout 2063 ms). Trace:
W20260806 04:01:23.535472 32475 rpcz_store.cc:269] 0806 04:01:21.591226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.595300 (+ 4074us) service_pool.cc:227] Handling call
0806 04:01:23.509148 (+1913848us) spinlock_profiling.cc:243] Waited 88.5 ms on lock 0x7b54000497d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:23.513681 (+ 4533us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":203596160}
I20260806 04:01:23.472801 3811 raft_consensus.cc:686] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:23.533519 32603 raft_consensus.cc:2379] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:23.515638 32604 rpcz_store.cc:276] 0806 04:01:21.482551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.482690 (+ 139us) service_pool.cc:227] Handling call
0806 04:01:23.488506 (+2005816us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400306954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d25cf2a 00007fd172199c6e 00007fd16b890807 00007fd16b890792 00007fd16b8906c1 00007fd16b89063c 00007fd16b88f8f2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:23.494767 (+ 6261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":538141440}
W20260806 04:01:23.536044 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 75142.2us
I20260806 04:01:23.391470 3854 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.549836 1136 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:23.510628 3794 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:23.561548 32609 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:23.573844 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.574615 1185 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:23.576434 32427 leader_election.cc:302] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:23.582590 1183 raft_consensus.cc:1226] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 04:01:23.584003 3800 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:23.584538 3800 raft_consensus.cc:686] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:23.585842 3800 consensus_queue.cc:237] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:23.588547 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:23.597038 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:23.601775 32551 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:23.565254 3855 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:23.636142 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.637530 1185 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:23.577013 3794 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:23.649770 32551 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:23.652513 1186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 673) took 1760 ms (client timeout 2239 ms). Trace:
W20260806 04:01:23.652940 1186 rpcz_store.cc:269] 0806 04:01:21.892469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:22.131494 (+239025us) service_pool.cc:227] Handling call
0806 04:01:23.652345 (+1520851us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54007f0154. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:23.652450 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2522436480}
W20260806 04:01:23.641616 32559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.145s user 0.012s sys 0.052s
I20260806 04:01:23.644194 3756 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:23.645589 3832 consensus_queue.cc:1046] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20260806 04:01:23.646808 3636 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.607124 3779 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:23.654135 32427 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
W20260806 04:01:23.654633 32559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.210s user 0.013s sys 0.052s
I20260806 04:01:23.655617 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.667815 32603 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:23.640552 3855 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:23.636281 3725 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:23.664700 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.670358 32427 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:23.637593 3582 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.543306 3891 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.678264 3891 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.671646 3725 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:23.657338 3793 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:23.681617 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:23.679947 3725 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:23.672773 3855 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:23.680274 3711 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:23.680737 3636 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:23.684157 3793 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:23.691565 3697 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:23.692337 3829 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:23.706163 3724 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.018s user 0.028s sys 0.030s
I20260806 04:01:23.693259 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:23.726081 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.681177 3283 consensus_queue.cc:1046] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260806 04:01:23.672003 3779 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.729490 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.701887 3791 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:23.707808 3711 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:23.710755 3636 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:23.720911 3793 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:23.667513 3811 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.719825 3697 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.690136 3582 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.762439 3572 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:23.739596 3569 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.315s user 0.021s sys 0.043s
I20260806 04:01:23.736128 1183 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:23.766314 3572 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.737753 3711 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:01:23.732337 3772 consensus_meta.cc:287] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.506s user 0.014s sys 0.000s
I20260806 04:01:23.739902 3636 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:23.764055 3697 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:23.729386 3829 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.794539 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:23.809726 32475 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:23.727849 3724 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.043s user 0.028s sys 0.030s
W20260806 04:01:23.766309 3569 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.411s user 0.021s sys 0.043s
I20260806 04:01:23.821496 3808 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.822140 3808 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.824440 3808 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.786131 3762 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:23.737753 3791 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:23.834090 1139 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.833s user 0.016s sys 0.028s
W20260806 04:01:23.887070 1139 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.886s user 0.016s sys 0.028s
I20260806 04:01:23.883744 3808 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.889034 3808 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:23.830987 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.832s user 0.021s sys 0.032s
W20260806 04:01:23.884099 1188 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.684s user 0.029s sys 0.030s
W20260806 04:01:23.892170 1188 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.692s user 0.029s sys 0.030s
W20260806 04:01:23.895454 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.893s user 0.021s sys 0.032s
I20260806 04:01:23.905344 3711 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.907722 3711 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.891109 3808 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.927687 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:23.928748 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.874233 3283 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.799646 3772 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:23.917831 32475 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:23.922987 3582 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.923295 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:23.925448 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.925699 3452 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:23.926136 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:23.936679 32682 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.909s user 0.007s sys 0.026s
I20260806 04:01:23.872737 3762 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:23.943697 32682 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.916s user 0.007s sys 0.026s
I20260806 04:01:23.944729 32470 raft_consensus.cc:2379] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 2: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:23.908924 32746 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.924s user 0.028s sys 0.031s
I20260806 04:01:23.931136 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:23.945994 32745 raft_consensus.cc:2379] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 2: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:23.932886 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.893221 3905 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:23.935819 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:23.953531 32475 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:23.884212 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:23.834156 3756 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:23.946561 32683 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:23.947177 32746 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.962s user 0.028s sys 0.032s
I20260806 04:01:23.943704 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:23.944481 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 1.09212e+06us
W20260806 04:01:23.953944 32429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.011s sys 0.031s
I20260806 04:01:23.927532 3725 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:23.963191 32429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.937s user 0.011s sys 0.031s
I20260806 04:01:23.964941 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.966226 32469 raft_consensus.cc:1226] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:23.968801 32603 raft_consensus.cc:1226] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:23.976922 32475 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:23.977619 32603 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:23.970935 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:23.979331 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:23.967344 3582 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:23.994828 3582 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.997237 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:23.997502 3582 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.997895 32745 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:23.998073 3582 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:23.999357 3582 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:23.999883 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:24.000608 32745 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:24.001344 32688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.013s sys 0.039s
W20260806 04:01:24.001718 32688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.011s user 0.014s sys 0.039s
I20260806 04:01:24.002573 3913 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:24.003103 3756 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:24.003244 3913 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:23.972142 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:24.003660 3756 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:23.968506 3779 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.953158 3452 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:24.005738 3756 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:24.003140 3572 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:24.006587 3452 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:24.007762 3913 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:23.938324 3283 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:24.002096 3582 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:24.009543 32604 raft_consensus.cc:1226] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:24.010915 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:24.000309 32558 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.878s user 0.014s sys 0.037s
I20260806 04:01:23.959285 3905 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:24.012072 32745 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
W20260806 04:01:24.081364 32610 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.781s user 0.013s sys 0.023s
W20260806 04:01:24.063258 32428 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.028s user 0.009s sys 0.028s
W20260806 04:01:24.107324 32466 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.005s sys 0.033s
I20260806 04:01:24.065181 32477 raft_consensus.cc:1226] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: []
W20260806 04:01:24.104734 32615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.896s user 0.000s sys 0.018s
W20260806 04:01:24.055600 1137 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.060s user 0.014s sys 0.023s
I20260806 04:01:24.062741 32603 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 04:01:24.428259 1137 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.433s user 0.015s sys 0.040s
W20260806 04:01:24.075443 32474 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.010s sys 0.027s
W20260806 04:01:24.015494 32558 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.097s user 0.015s sys 0.037s
W20260806 04:01:24.429243 32474 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.269s user 0.017s sys 0.035s
I20260806 04:01:24.013022 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:24.255465 32739 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.966s user 0.003s sys 0.024s
W20260806 04:01:24.430964 32739 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.142s user 0.004s sys 0.030s
W20260806 04:01:24.285024 32614 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.937s user 0.011s sys 0.015s
W20260806 04:01:24.432008 32614 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.094s user 0.017s sys 0.016s
W20260806 04:01:24.427711 32615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.219s user 0.009s sys 0.025s
W20260806 04:01:24.426223 32428 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.391s user 0.017s sys 0.040s
W20260806 04:01:24.428689 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 1.43322e+06us
I20260806 04:01:24.437922 32553 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
W20260806 04:01:24.425765 32610 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.125s user 0.019s sys 0.029s
W20260806 04:01:24.427712 32466 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.201s user 0.010s sys 0.041s
W20260806 04:01:24.375131 1192 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.900s user 0.005s sys 0.020s
W20260806 04:01:24.522924 1192 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.048s user 0.005s sys 0.020s
W20260806 04:01:24.434665 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.40018e+06us
W20260806 04:01:24.308899 32608 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.005s sys 0.005s
W20260806 04:01:24.533973 32608 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.891s user 0.005s sys 0.006s
W20260806 04:01:24.234961 3434 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.063s user 0.001s sys 0.038s
W20260806 04:01:24.120429 3274 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.033s user 0.013s sys 0.007s
W20260806 04:01:24.125880 3693 consensus_meta.cc:287] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.956s user 0.010s sys 0.003s
W20260806 04:01:24.145174 3303 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.008s sys 0.017s
W20260806 04:01:24.237064 3704 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.993s user 0.009s sys 0.017s
W20260806 04:01:24.423758 3855 consensus_meta.cc:287] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.706s user 0.004s sys 0.002s
W20260806 04:01:24.122298 3174 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.006s sys 0.013s
I20260806 04:01:24.013834 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:24.395694 3932 log.cc:924] Time spent T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.160s user 0.000s sys 0.000s
W20260806 04:01:24.248925 3818 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.202s user 0.006s sys 0.029s
W20260806 04:01:24.377058 3882 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.957s user 0.013s sys 0.027s
I20260806 04:01:24.004314 3766 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:24.244848 3790 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.024s user 0.010s sys 0.013s
I20260806 04:01:24.012470 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:23.972524 3711 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:25.275254 32683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.001s sys 0.000s
W20260806 04:01:25.239310 1186 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.510s user 0.042s sys 0.088s
W20260806 04:01:25.408432 1186 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.679s user 0.043s sys 0.101s
W20260806 04:01:24.158433 3916 log.cc:924] Time spent T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.322s user 0.001s sys 0.000s
W20260806 04:01:25.383664 32551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.001s sys 0.009s
W20260806 04:01:24.083957 3802 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.044s user 0.009s sys 0.021s
W20260806 04:01:25.079698 32473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.536s user 0.002s sys 0.027s
W20260806 04:01:25.390100 3303 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.936s user 0.022s sys 0.066s
W20260806 04:01:24.884464 32478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.183s user 0.000s sys 0.021s
W20260806 04:01:25.087086 1188 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.174s user 0.005s sys 0.040s
W20260806 04:01:25.435140 1188 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.522s user 0.005s sys 0.045s
W20260806 04:01:25.261699 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.000s sys 0.012s
W20260806 04:01:25.435603 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.896s user 0.000s sys 0.016s
W20260806 04:01:25.384071 32615 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.820s user 0.004s sys 0.005s
I20260806 04:01:24.645548 3582 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:25.405887 3836 raft_consensus.cc:3032] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:25.448113 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 777) took 3591 ms. Trace:
W20260806 04:01:25.205909 1192 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.001s sys 0.001s
W20260806 04:01:25.244871 32746 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.283s user 0.018s sys 0.029s
W20260806 04:01:25.274544 32739 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.843s user 0.002s sys 0.012s
W20260806 04:01:25.384369 32474 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.844s user 0.003s sys 0.007s
W20260806 04:01:25.284508 32688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.009s sys 0.001s
W20260806 04:01:24.885375 32427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.138s user 0.008s sys 0.038s
W20260806 04:01:25.245146 1189 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.599s user 0.000s sys 0.031s
I20260806 04:01:24.417711 3933 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:24.418277 3936 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.397024 32470 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
W20260806 04:01:24.308689 1184 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.002s sys 0.010s
W20260806 04:01:25.378634 32430 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.166s user 0.017s sys 0.082s
I20260806 04:01:25.274582 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 549) took 1632 ms. Trace:
W20260806 04:01:25.453671 32430 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.474s user 0.019s sys 0.082s
I20260806 04:01:24.124969 3829 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:24.416850 3283 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:24.143067 3854 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:24.532850 3590 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:24.374990 3780 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.003s sys 0.025s
W20260806 04:01:25.275794 3814 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.238s user 0.019s sys 0.020s
W20260806 04:01:25.408233 32683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.770s user 0.011s sys 0.006s
W20260806 04:01:25.284256 3811 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.875s user 0.003s sys 0.007s
I20260806 04:01:24.261173 3791 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:24.643998 3697 consensus_meta.cc:287] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.825s user 0.005s sys 0.004s
I20260806 04:01:24.145612 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:24.563982 3434 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.463s user 0.006s sys 0.046s
W20260806 04:01:25.382300 3704 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.139s user 0.024s sys 0.064s
W20260806 04:01:24.783385 1183 consensus_meta.cc:287] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.983s user 0.008s sys 0.000s
W20260806 04:01:24.752643 3794 consensus_meta.cc:287] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 1.091s user 0.008s sys 0.004s
W20260806 04:01:25.392445 3882 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.222s user 0.019s sys 0.029s
W20260806 04:01:25.086639 3800 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.192s user 0.002s sys 0.042s
I20260806 04:01:25.435829 3711 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.468888 3936 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:01:25.425352 32551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.862s user 0.001s sys 0.009s
W20260806 04:01:24.648037 3647 consensus_meta.cc:287] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 1.222s user 0.012s sys 0.001s
I20260806 04:01:25.497591 3647 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.498420 3936 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:25.170018 3569 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.287s user 0.000s sys 0.028s
W20260806 04:01:25.499118 3569 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.618s user 0.001s sys 0.034s
W20260806 04:01:25.428992 32473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.885s user 0.003s sys 0.031s
I20260806 04:01:23.971299 3782 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 04:01:25.396555 3274 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.309s user 0.027s sys 0.021s
W20260806 04:01:25.390367 3174 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.124s user 0.040s sys 0.049s
W20260806 04:01:24.781738 3636 consensus_meta.cc:287] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.966s user 0.007s sys 0.002s
W20260806 04:01:25.397383 3818 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.344s user 0.008s sys 0.041s
I20260806 04:01:25.500514 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1823) took 2337 ms. Trace:
I20260806 04:01:24.651939 3939 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.500905 32466 rpcz_store.cc:276] 0806 04:01:23.163374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.163649 (+ 275us) service_pool.cc:227] Handling call
0806 04:01:23.164099 (+ 450us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:23.164438 (+ 339us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:23.164487 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:23.170448 (+ 5961us) log.cc:842] Serialized 26 byte log entry
0806 04:01:24.512890 (+1342442us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:24.512935 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:24.513022 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:25.500288 (+987266us) raft_consensus.cc:1671] finished
0806 04:01:25.500307 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.500436 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3625,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":17315,"thread_start_us":1285547,"threads_started":1,"wal-append.queue_time_us":2293887}
W20260806 04:01:25.395807 3790 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.175s user 0.010s sys 0.020s
I20260806 04:01:24.557562 3772 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:25.436142 32615 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.873s user 0.005s sys 0.005s
W20260806 04:01:25.124470 1139 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.230s user 0.012s sys 0.016s
W20260806 04:01:25.503556 1139 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.609s user 0.017s sys 0.017s
I20260806 04:01:25.505069 3830 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.505577 3830 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:24.650707 3905 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:25.436831 32478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.736s user 0.011s sys 0.022s
I20260806 04:01:24.555764 3913 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:25.361083 3926 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:25.508389 1190 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:25.508200 3913 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.509364 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:25.510386 3913 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:25.510624 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:25.511270 3782 raft_consensus.cc:686] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:24.645663 3808 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:25.512583 3830 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:25.512369 3808 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:25.512243 3782 consensus_queue.cc:237] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:25.514346 3808 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:24.652312 3762 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:24.648360 3870 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:25.499861 3647 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:25.514560 32741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2143) took 3410 ms (client timeout 1680 ms). Trace:
W20260806 04:01:25.517360 32741 rpcz_store.cc:269] 0806 04:01:22.103714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:22.137561 (+ 33847us) service_pool.cc:227] Handling call
0806 04:01:25.514357 (+3376796us) spinlock_profiling.cc:243] Waited 3.35 s on lock 0x7b54002120d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:25.514483 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7710465792}
I20260806 04:01:25.517791 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1833) took 2013 ms. Trace:
I20260806 04:01:25.518218 32473 rpcz_store.cc:276] 0806 04:01:23.504009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.508899 (+ 4890us) service_pool.cc:227] Handling call
0806 04:01:23.541353 (+ 32454us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:23.541593 (+ 240us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:23.541616 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:23.543712 (+ 2096us) log.cc:842] Serialized 26 byte log entry
0806 04:01:25.517458 (+1973746us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:25.517506 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.517577 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:25.517618 (+ 41us) raft_consensus.cc:1671] finished
0806 04:01:25.517628 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.517733 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1973262,"threads_started":1,"wal-append.queue_time_us":1971854}
I20260806 04:01:25.519577 3974 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:25.520125 3974 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.522065 3974 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:25.500628 3636 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.525897 3636 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:25.526543 32467 raft_consensus.cc:1226] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:25.529021 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1838) took 1697 ms. Trace:
I20260806 04:01:24.068117 3793 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:25.210623 32682 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.001s sys 0.001s
I20260806 04:01:25.396263 3756 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:25.487855 3800 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.606s user 0.002s sys 0.049s
W20260806 04:01:25.432027 3802 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.389s user 0.009s sys 0.030s
I20260806 04:01:25.453773 32608 rpcz_store.cc:276] 0806 04:01:23.641987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.642134 (+ 147us) service_pool.cc:227] Handling call
0806 04:01:23.642464 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:23.642662 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:23.643035 (+ 373us) log.cc:842] Serialized 14 byte log entry
0806 04:01:25.274116 (+1631081us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:25.274195 (+ 79us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:25.274267 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.274358 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.274518 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":287228,"prepare.queue_time_us":234,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"spinlock_wait_cycles":128128,"thread_start_us":1343168,"threads_started":2}
W20260806 04:01:25.449637 1192 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.926s user 0.002s sys 0.009s
W20260806 04:01:25.449757 32746 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.488s user 0.021s sys 0.031s
I20260806 04:01:25.545147 32746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 417) took 3244 ms. Trace:
W20260806 04:01:25.449872 32739 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.018s user 0.004s sys 0.013s
I20260806 04:01:25.511737 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:25.450289 32688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.902s user 0.009s sys 0.004s
W20260806 04:01:25.450176 32474 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.910s user 0.003s sys 0.008s
I20260806 04:01:25.451011 1188 rpcz_store.cc:276] 0806 04:01:21.856693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.856839 (+ 146us) service_pool.cc:227] Handling call
0806 04:01:21.888603 (+ 31764us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:21.888937 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:21.888976 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:21.889418 (+ 442us) log.cc:842] Serialized 26 byte log entry
0806 04:01:22.164258 (+274840us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:23.892524 (+1728266us) log.cc:842] Serialized 14 byte log entry
0806 04:01:25.442266 (+1549742us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.442353 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:25.447922 (+ 5569us) raft_consensus.cc:1671] finished
0806 04:01:25.447937 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.448049 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45958,"prepare.queue_time_us":1720540,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":57829,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":26542,"thread_start_us":3436235,"threads_started":3,"wal-append.queue_time_us":1089562,"wal-append.run_cpu_time_us":3955,"wal-append.run_wall_time_us":1998993}
I20260806 04:01:25.547691 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1820) took 2402 ms. Trace:
W20260806 04:01:25.450430 32427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.705s user 0.010s sys 0.051s
W20260806 04:01:25.450565 1189 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.805s user 0.000s sys 0.035s
I20260806 04:01:25.548444 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:25.474366 3832 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:25.448359 3836 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:25.458722 1184 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.809s user 0.008s sys 0.013s
I20260806 04:01:25.403036 3766 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.475311 32610 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1799) took 4092 ms. Trace:
I20260806 04:01:25.468957 3590 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:25.459998 3367 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:25.463233 3933 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.477901 32614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 542) took 2199 ms. Trace:
W20260806 04:01:25.465296 3780 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.899s user 0.009s sys 0.043s
I20260806 04:01:25.477414 4007 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:25.452880 3744 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:25.470346 3814 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.486s user 0.020s sys 0.025s
I20260806 04:01:25.449024 3995 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:25.472559 3811 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.060s user 0.003s sys 0.009s
I20260806 04:01:25.452600 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1820) took 1440 ms. Trace:
I20260806 04:01:25.451169 3261 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:25.462664 3283 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:25.164520 3911 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.233s user 0.015s sys 0.015s
I20260806 04:01:25.496423 32470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59514 (request call id 1840) took 1550 ms. Trace:
I20260806 04:01:25.507131 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:24.912242 3725 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:25.468847 3751 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.468853 3854 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:25.530838 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1835) took 1847 ms. Trace:
W20260806 04:01:25.537678 32682 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.973s user 0.010s sys 0.001s
W20260806 04:01:25.261286 1136 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.301s user 0.007s sys 0.041s
I20260806 04:01:25.529327 32467 rpcz_store.cc:276] 0806 04:01:23.831334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.831533 (+ 199us) service_pool.cc:227] Handling call
0806 04:01:23.880330 (+ 48797us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:25.526923 (+1646593us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.528740 (+ 1817us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540009db54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:25.528959 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3785493760}
I20260806 04:01:25.533061 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1813) took 2346 ms. Trace:
I20260806 04:01:25.408340 32612 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:25.400007 3855 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.548023 32474 rpcz_store.cc:276] 0806 04:01:23.145087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.145236 (+ 149us) service_pool.cc:227] Handling call
0806 04:01:23.145573 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:23.145782 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:01:24.540155 (+1394373us) log.cc:842] Serialized 14 byte log entry
0806 04:01:25.547285 (+1007130us) log.cc:842] Serialized 14 byte log entry
0806 04:01:25.547405 (+ 120us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:25.547438 (+ 33us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:25.547483 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.547542 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.547642 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":2284351,"threads_started":2}
I20260806 04:01:25.550891 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 787) took 1983 ms. Trace:
I20260806 04:01:25.544304 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 759) took 2085 ms. Trace:
I20260806 04:01:25.545475 32746 rpcz_store.cc:276] 0806 04:01:22.300452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:22.354237 (+ 53785us) service_pool.cc:227] Handling call
0806 04:01:22.354582 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:22.984968 (+630386us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:23.961259 (+976291us) log.cc:842] Serialized 14 byte log entry
0806 04:01:25.544804 (+1583545us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:25.544862 (+ 58us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:25.544909 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.544982 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.545086 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1664306,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"thread_start_us":2559024,"threads_started":2,"wal-append.queue_time_us":1554311}
I20260806 04:01:25.458552 3582 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:25.563000 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:25.563731 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:25.565594 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:25.530514 3434 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.498438 32740 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 04:01:25.565836 1190 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 04:01:25.480406 3791 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:25.494655 3693 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:25.530606 3905 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:25.568142 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:25.551447 3951 log.cc:924] Time spent T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20260806 04:01:25.568246 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2146) took 2534 ms. Trace:
I20260806 04:01:25.566557 32610 rpcz_store.cc:276] 0806 04:01:21.383022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:21.393774 (+ 10752us) service_pool.cc:227] Handling call
0806 04:01:21.402427 (+ 8653us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:21.402774 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:21.402803 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:21.403261 (+ 458us) log.cc:842] Serialized 52 byte log entry
0806 04:01:22.300556 (+897295us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b180042e5c0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb411 00007fd16d2285cf
0806 04:01:24.441915 (+2141359us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:24.441956 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:24.442035 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:25.474979 (+1032944us) raft_consensus.cc:1671] finished
0806 04:01:25.475003 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.475244 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4958,"spinlock_wait_cycles":2510208,"thread_start_us":2141153,"threads_started":1,"wal-append.queue_time_us":4044690}
I20260806 04:01:25.485363 1183 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:25.539006 3379 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.539387 3839 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:25.484272 3697 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.515372 3870 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.488230 3794 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:25.488555 32737 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:25.551971 3218 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:25.934382 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:25.925937 1136 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.966s user 0.007s sys 0.053s
I20260806 04:01:25.924901 32470 rpcz_store.cc:276] 0806 04:01:23.946118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.954184 (+ 8066us) service_pool.cc:227] Handling call
0806 04:01:25.496216 (+1542032us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54002026d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:25.496356 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3259937536}
I20260806 04:01:25.924393 32605 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
W20260806 04:01:25.927395 32559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.005s sys 0.020s
I20260806 04:01:25.925639 1189 rpcz_store.cc:276] 0806 04:01:23.567511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.567751 (+ 240us) service_pool.cc:227] Handling call
0806 04:01:23.568137 (+ 386us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:23.568505 (+ 368us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:23.568528 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:23.591235 (+ 22707us) log.cc:842] Serialized 26 byte log entry
0806 04:01:25.550570 (+1959335us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:25.550609 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.550687 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:25.550730 (+ 43us) raft_consensus.cc:1671] finished
0806 04:01:25.550740 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.550837 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":449,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":17155,"thread_start_us":1904561,"threads_started":1,"wal-append.queue_time_us":1921516}
I20260806 04:01:25.551342 3913 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:25.604326 32429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.249s user 0.043s sys 0.055s
I20260806 04:01:25.926586 32739 rpcz_store.cc:276] 0806 04:01:23.033679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.035347 (+ 1668us) service_pool.cc:227] Handling call
0806 04:01:23.038681 (+ 3334us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:23.262877 (+224196us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:24.431362 (+1168485us) log.cc:842] Serialized 14 byte log entry
0806 04:01:25.546577 (+1115215us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:25.546626 (+ 49us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:25.546674 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.567996 (+ 21322us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54000bde54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:25.568037 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.568180 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1388759,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":2163,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":16276,"spinlock_wait_cycles":515001088,"thread_start_us":2257115,"threads_started":2,"wal-append.queue_time_us":1012475}
W20260806 04:01:25.944839 32429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.590s user 0.049s sys 0.060s
W20260806 04:01:25.937386 32559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.571s user 0.006s sys 0.020s
W20260806 04:01:25.928205 32558 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.017s sys 0.007s
I20260806 04:01:25.931008 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:25.963244 32558 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.717s user 0.018s sys 0.007s
I20260806 04:01:25.925881 1192 rpcz_store.cc:276] 0806 04:01:23.458796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.465065 (+ 6269us) service_pool.cc:227] Handling call
0806 04:01:23.465396 (+ 331us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:23.465607 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:24.523650 (+1058043us) log.cc:842] Serialized 14 byte log entry
0806 04:01:25.543948 (+1020298us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:25.544002 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:25.544047 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.544139 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.544246 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2983,"prepare.queue_time_us":534845,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":70,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":16495,"spinlock_wait_cycles":12039936,"thread_start_us":2068012,"threads_started":2,"wal-append.queue_time_us":1042903}
I20260806 04:01:25.924265 32478 rpcz_store.cc:276] 0806 04:01:23.683199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.683355 (+ 156us) service_pool.cc:227] Handling call
0806 04:01:23.700329 (+ 16974us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:23.700564 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:23.700588 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:23.701086 (+ 498us) log.cc:842] Serialized 26 byte log entry
0806 04:01:25.507773 (+1806687us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:25.507815 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.507884 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:25.530605 (+ 22721us) raft_consensus.cc:1671] finished
0806 04:01:25.530626 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.530771 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1806406,"threads_started":1,"wal-append.queue_time_us":1808401}
I20260806 04:01:25.944162 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:25.716120 3966 log.cc:924] Time spent T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
W20260806 04:01:25.925128 3911 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.034s user 0.023s sys 0.032s
W20260806 04:01:25.934823 3945 log.cc:924] Time spent T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.368s user 0.000s sys 0.014s
W20260806 04:01:25.712656 3991 log.cc:924] Time spent T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
W20260806 04:01:25.927551 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.365s user 0.015s sys 0.012s
W20260806 04:01:25.955440 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.009s sys 0.010s
I20260806 04:01:25.945365 1137 leader_election.cc:398] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
W20260806 04:01:25.943717 32612 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1822) took 1919 ms (client timeout 1564 ms). Trace:
I20260806 04:01:25.966022 1187 raft_consensus.cc:2354] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:25.966288 32471 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:25.620296 3926 raft_consensus.cc:3032] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:25.926198 32614 rpcz_store.cc:276] 0806 04:01:23.278617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.278852 (+ 235us) service_pool.cc:227] Handling call
0806 04:01:23.279213 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:23.279526 (+ 313us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:23.279551 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:23.282710 (+ 3159us) log.cc:842] Serialized 26 byte log entry
0806 04:01:24.432448 (+1149738us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:24.432490 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:24.432561 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:25.477683 (+1045122us) raft_consensus.cc:1671] finished
0806 04:01:25.477701 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.477822 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":126316,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"thread_start_us":1094662,"threads_started":1,"wal-append.queue_time_us":2044885}
I20260806 04:01:25.924592 32615 rpcz_store.cc:276] 0806 04:01:23.186195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.186308 (+ 113us) service_pool.cc:227] Handling call
0806 04:01:23.195562 (+ 9254us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:23.205429 (+ 9867us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:23.205462 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:23.205918 (+ 456us) log.cc:842] Serialized 26 byte log entry
0806 04:01:24.433922 (+1228004us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:25.503316 (+1069394us) log.cc:842] Serialized 14 byte log entry
0806 04:01:25.532762 (+ 29446us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:25.532838 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:25.532878 (+ 40us) raft_consensus.cc:1671] finished
0806 04:01:25.532890 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:25.533002 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40573,"prepare.queue_time_us":1244464,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":56694,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":11184,"thread_start_us":2164651,"threads_started":2,"wal-append.queue_time_us":2251733}
I20260806 04:01:25.963799 3725 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:25.932704 3636 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.551075 3756 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:25.988058 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.009s sys 0.010s
I20260806 04:01:25.988212 1137 leader_election.cc:401] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:25.929268 3711 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:25.987988 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.425s user 0.015s sys 0.012s
W20260806 04:01:26.016206 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.57826e+06us
I20260806 04:01:26.016988 32553 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:25.985914 3984 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.985677 32682 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:26.023723 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:25.969676 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:25.564215 3832 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.933033 3793 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:25.944583 4003 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:25.943915 1183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 763) took 2284 ms (client timeout 1677 ms). Trace:
W20260806 04:01:25.955533 32737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1682) took 1886 ms (client timeout 1805 ms). Trace:
I20260806 04:01:25.989131 1182 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 7: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:25.933225 3379 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:25.932745 3839 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.039930 4003 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:26.022796 3711 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:26.016590 3636 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.052960 3711 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.053807 4003 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:26.054430 3839 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:25.994692 3772 consensus_queue.cc:1035] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260806 04:01:26.032819 3808 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:26.054931 3711 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.055398 3808 raft_consensus.cc:686] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:26.055648 3590 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.056499 3582 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:25.967933 1189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:26.013804 3725 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.057372 1189 raft_consensus.cc:595] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:26.058460 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:26.058851 1187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 789) took 2478 ms (client timeout 1624 ms). Trace:
W20260806 04:01:26.059252 1187 rpcz_store.cc:269] 0806 04:01:23.580204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.588283 (+ 8079us) service_pool.cc:227] Handling call
0806 04:01:26.058577 (+2470294us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b5400710554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:26.058782 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5465822592}
I20260806 04:01:25.935549 3995 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:26.022287 1186 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.012s sys 0.044s
I20260806 04:01:25.933645 3434 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:25.984750 4006 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:26.065560 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:26.065930 4006 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.025488 32553 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.051565 3814 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:25.924805 32602 rpcz_store.cc:276] 0806 04:01:24.012033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:24.012193 (+ 160us) service_pool.cc:227] Handling call
0806 04:01:25.449456 (+1437263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:25.933504 3854 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.054795 3636 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:25.552585 3939 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.078608 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.089460 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:26.138459 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 203518us
W20260806 04:01:26.142542 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 117091us
I20260806 04:01:25.935693 3283 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.936126 3791 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.935806 3744 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.933991 3367 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.936345 3261 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.000463 3926 raft_consensus.cc:729] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:25.996016 3766 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:25.944113 3905 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.961493 3693 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:25.933748 4007 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.201018 3693 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:26.033141 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 371159us
I20260806 04:01:26.036152 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:26.038344 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 339513us
I20260806 04:01:26.201861 1192 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:01:26.015780 3790 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.011s sys 0.005s
I20260806 04:01:25.945501 3751 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:25.974256 3913 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:25.609691 3836 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:25.990996 32612 rpcz_store.cc:269] 0806 04:01:24.013442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:24.013591 (+ 149us) service_pool.cc:227] Handling call
0806 04:01:25.929437 (+1915846us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:26.048133 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:26.039232 32737 rpcz_store.cc:269] 0806 04:01:24.059906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:24.062278 (+ 2372us) service_pool.cc:227] Handling call
0806 04:01:25.946205 (+1883927us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:26.050472 3782 consensus_queue.cc:1046] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:26.220737 3782 raft_consensus.cc:3032] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:26.221105 3782 raft_consensus.cc:729] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:26.221669 4019 log.cc:924] Time spent T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.084s user 0.002s sys 0.000s
I20260806 04:01:26.221908 3782 consensus_queue.cc:260] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.223125 3782 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:26.223541 1188 raft_consensus.cc:1226] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 04:01:26.050961 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.058669 3725 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.225935 3751 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.059876 1189 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:26.027070 3174 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:26.062871 32471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1582) took 2119 ms (client timeout 1803 ms). Trace:
I20260806 04:01:26.029716 3984 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.228364 3174 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.233800 3434 leader_election.cc:288] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.196440 3808 consensus_queue.cc:237] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.235491 3174 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.036743 3303 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:26.236136 3303 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:26.236518 3303 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:26.203029 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.237376 32740 raft_consensus.cc:2379] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:26.228461 3832 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:25.844707 3941 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.162s
I20260806 04:01:25.967279 4000 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:26.241420 3913 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:26.241228 4000 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:26.242460 4025 log.cc:924] Time spent T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 04:01:26.243335 4000 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.029503 4008 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 04:01:26.068495 1186 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s user 0.012s sys 0.044s
I20260806 04:01:26.248404 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 790) took 2532 ms. Trace:
W20260806 04:01:26.059367 3976 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.364s user 0.000s sys 0.002s
I20260806 04:01:26.248729 1186 rpcz_store.cc:276] 0806 04:01:23.715715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.715855 (+ 140us) service_pool.cc:227] Handling call
0806 04:01:23.716191 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:23.716420 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 04:01:25.408967 (+1692547us) log.cc:842] Serialized 14 byte log entry
0806 04:01:26.248070 (+839103us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:26.248138 (+ 68us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:26.248181 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:26.248250 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:26.248349 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":284474,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"thread_start_us":2518529,"threads_started":2,"wal-append.queue_time_us":749369}
I20260806 04:01:26.039921 3974 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:26.024648 32739 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:26.056435 3756 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:26.144891 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 71673.9us
I20260806 04:01:26.249066 1192 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:26.143153 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:26.250407 3974 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.251123 1185 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.252347 3569 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:26.252859 3569 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.160848 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.089557 3590 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:01:26.033969 1183 rpcz_store.cc:269] 0806 04:01:23.648272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.656351 (+ 8079us) service_pool.cc:227] Handling call
0806 04:01:25.932866 (+2276515us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:26.256093 32741 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:26.256776 3995 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:26.258057 3283 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.259533 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.259776 3939 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.203581 3800 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:26.072638 4018 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:26.073386 3582 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:26.188617 32428 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.156466 3698 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
W20260806 04:01:26.160785 3967 log.cc:924] Time spent T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.315s user 0.001s sys 0.000s
I20260806 04:01:26.190932 32746 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 04:01:26.274230 4027 log.cc:924] Time spent T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 04:01:26.274655 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.190754 1137 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.274952 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.275310 32612 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:26.210975 4016 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:26.276028 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 85358.3us
I20260806 04:01:26.276449 4016 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.277207 32466 raft_consensus.cc:1226] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:26.091137 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 764) took 2442 ms. Trace:
I20260806 04:01:26.197211 1190 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:26.276180 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:26.198202 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.198585 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:26.177600 32738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:01:26.280498 32738 raft_consensus.cc:595] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:26.082427 3814 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.281596 3744 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:26.191373 4002 log.cc:924] Time spent T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.200s user 0.002s sys 0.000s
I20260806 04:01:26.281911 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.212695 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.276451 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:26.205883 3790 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.011s sys 0.005s
I20260806 04:01:26.196622 3697 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.201313 4007 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.243197 3926 consensus_queue.cc:260] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.276136 4018 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.284945 3926 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:25.966606 32694 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.016s sys 0.016s
W20260806 04:01:26.286684 32694 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.902s user 0.016s sys 0.016s
I20260806 04:01:26.244866 4006 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:26.228718 32471 rpcz_store.cc:269] 0806 04:01:23.929114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.929239 (+ 125us) service_pool.cc:227] Handling call
0806 04:01:26.048402 (+2119163us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b540009d654. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:26.048538 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4608052608}
I20260806 04:01:26.246608 3379 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.246973 3794 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:26.146953 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:26.300108 3697 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:26.309902 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 163010us
I20260806 04:01:26.178035 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.260674 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:26.310842 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.311201 32553 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.282989 3814 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.260834 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.067122 3711 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:26.313037 3974 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.313489 3711 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.315577 3711 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.312886 4007 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:26.258998 1185 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:26.261437 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.324541 1192 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:26.211602 3766 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:26.324827 32683 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:26.330654 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.331754 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.265844 3800 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.231622 3870 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:26.277079 4015 log.cc:924] Time spent T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 04:01:26.289482 1184 rpcz_store.cc:276] 0806 04:01:23.648541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:23.648691 (+ 150us) service_pool.cc:227] Handling call
0806 04:01:23.649108 (+ 417us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:23.649411 (+ 303us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:01:25.575400 (+1925989us) log.cc:842] Serialized 14 byte log entry
0806 04:01:26.061992 (+486592us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:26.062944 (+ 952us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:26.072773 (+ 9829us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:26.074848 (+ 2075us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:26.081730 (+ 6882us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9459,"prepare.queue_time_us":351212,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":26071,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"thread_start_us":2347252,"threads_started":2,"wal-append.queue_time_us":561009}
I20260806 04:01:26.278972 32475 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:26.279193 32478 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:26.336709 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.337025 3367 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:26.337497 32478 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:26.338424 3779 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:26.338467 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.280575 3854 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:26.339360 32478 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:26.340278 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:26.340206 4043 consensus_queue.cc:1046] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:26.341032 32478 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.341547 4043 raft_consensus.cc:3032] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:26.341902 4043 raft_consensus.cc:729] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:26.342681 4043 consensus_queue.cc:260] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.279415 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.279942 3791 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.281140 32427 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:26.281692 4036 consensus_queue.cc:1046] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:26.274430 4041 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:26.286255 4018 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.244746 3905 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.189671 3636 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:26.194459 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 161179us
W20260806 04:01:26.292567 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 104038us
W20260806 04:01:26.299947 4023 log.cc:924] Time spent T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260806 04:01:26.234633 3984 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.197171 3855 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:26.317955 32553 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:26.358163 32553 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:26.359076 32553 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.360059 3997 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:26.360383 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.360641 3870 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:26.361042 3870 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:26.361474 3870 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:26.362092 3590 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:26.361055 32612 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 1.
I20260806 04:01:26.354841 4041 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.329691 4007 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:26.266135 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.339242 3913 consensus_queue.cc:1046] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260806 04:01:26.333362 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:26.333525 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.279959 3933 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.380138 32614 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.333534 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:26.345000 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 63931.2us
I20260806 04:01:26.343855 4043 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.345196 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.338657 32738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:01:26.350019 32478 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:26.351011 32475 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:26.392452 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.354779 3636 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:26.326821 1137 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:26.355762 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.329650 3698 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:26.369370 32553 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.370270 3790 raft_consensus.cc:3032] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:26.355713 32428 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.397712 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.394876 32683 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:26.345443 3779 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:26.350154 3711 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:26.400779 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.401010 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.401369 3711 raft_consensus.cc:686] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:26.401443 32612 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:26.402665 3905 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:26.402275 3711 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.375607 32615 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:26.329638 3379 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:26.404706 3379 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:26.405225 32738 raft_consensus.cc:595] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:26.405367 3367 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:26.405634 32738 raft_consensus.cc:603] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:26.406366 1187 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:26.379525 4007 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:26.407356 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.397856 3790 raft_consensus.cc:729] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:26.409088 3790 consensus_queue.cc:260] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.410300 3790 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.412544 32683 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:26.413117 4041 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.413338 3836 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:26.413632 3647 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:26.414053 4000 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:26.414376 3800 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.414942 3779 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:26.415284 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:26.416067 1184 raft_consensus.cc:2354] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:26.397686 3933 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:26.418787 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.419991 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.414125 3647 raft_consensus.cc:686] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:26.420768 32477 raft_consensus.cc:2396] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260806 04:01:26.421290 3647 consensus_queue.cc:237] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.421846 32477 raft_consensus.cc:1226] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:26.422148 32469 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:26.422825 3303 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.396293 3913 consensus_queue.cc:1225] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer ae0b3e43e7a64f789e51018fd5518d23 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:01:26.424026 32473 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:26.424291 3818 consensus_queue.cc:1046] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:26.405117 3379 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:26.365114 3997 raft_consensus.cc:686] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:26.426409 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260806 04:01:26.426947 3997 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.344353 3766 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.428424 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:26.386888 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.387001 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.387749 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.429888 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:26.388469 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.430575 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:26.389140 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.431203 1189 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:01:26.390118 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.390514 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.382485 32682 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.384703 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.393007 32475 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:26.430716 1185 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:26.392947 32427 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.395135 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.360301 3791 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20260806 04:01:26.396071 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.459589 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.332612 3582 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:26.397120 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 70382.5us
I20260806 04:01:26.379686 4016 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.379251 4042 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260806 04:01:26.405145 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:26.432300 32608 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:26.390715 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.503983 32474 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
W20260806 04:01:26.459947 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 67059.7us
I20260806 04:01:26.505760 32427 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.450505 32614 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:26.404582 3636 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.478868 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.496906 32602 raft_consensus.cc:2379] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:26.497398 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 115311us
I20260806 04:01:26.497532 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:26.507463 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.508713 32427 leader_election.cc:398] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:26.511145 3694 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:26.511168 3905 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:26.511691 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.511889 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.524519 32682 leader_election.cc:398] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5.
I20260806 04:01:26.525084 32682 leader_election.cc:401] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:26.525110 32739 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:26.429404 3367 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:26.458909 3913 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:26.513465 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.513597 3303 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.513702 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:26.551190 32749 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:26.516461 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:26.521440 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 125399us
I20260806 04:01:26.512357 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.523676 32427 leader_election.cc:401] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 04:01:26.523890 32474 raft_consensus.cc:2396] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 04:01:26.552500 32553 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.521039 32605 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:26.553722 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.501449 4000 raft_consensus.cc:686] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:26.556473 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 47890.1us
I20260806 04:01:26.559689 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.564525 32427 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:26.566675 32474 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:26.527336 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.524633 32471 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:26.499810 3870 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.530539 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.524776 3766 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.462947 3926 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:26.565310 3636 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.532682 32682 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:26.571759 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:26.505414 3800 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:26.547159 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.554788 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.575807 32478 raft_consensus.cc:2379] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:26.512145 4016 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:26.568473 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:26.580001 32748 raft_consensus.cc:2362] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2. Re-sending same reply.
I20260806 04:01:26.568864 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.532977 3367 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.525619 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.541260 3756 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5.
I20260806 04:01:26.574393 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.577157 32746 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:26.580780 32473 raft_consensus.cc:2379] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 2: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:26.560858 4008 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:26.555884 3434 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:26.646267 32553 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.537009 3794 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260806 04:01:26.578954 4003 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:26.653450 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.665282 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.666368 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.660703 32553 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.668009 32746 raft_consensus.cc:2396] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:01:26.579066 3800 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:26.580051 4041 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:26.580729 3974 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:26.670018 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.581286 3926 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:26.671458 32553 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.618745 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.676299 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.677368 32608 raft_consensus.cc:1226] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:26.681602 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.667943 4006 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.664462 4003 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:26.675118 4008 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:26.675433 3800 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:26.694396 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.699144 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.704221 32739 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:26.700429 32428 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.707140 32428 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.701773 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.701860 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.707866 32474 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:26.708669 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.678822 3974 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:26.664880 3818 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260806 04:01:26.711372 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.702549 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.669370 3933 consensus_queue.cc:1046] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260806 04:01:26.712047 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.712219 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.688316 3590 raft_consensus.cc:2741] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 04:01:26.697198 4003 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:26.689131 3926 raft_consensus.cc:686] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:26.691293 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.724565 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.727560 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.728147 32477 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:26.715131 32428 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.725356 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.703176 4008 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:26.728611 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.714157 3990 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:26.715870 3974 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:26.680878 4041 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.731874 32608 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:26.732681 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.732743 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:26.722596 32737 raft_consensus.cc:2379] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 2: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:26.720556 3994 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:26.724704 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.733901 4008 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:26.740439 4036 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:26.750339 32739 raft_consensus.cc:2396] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:01:26.738957 32428 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:26.742035 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.717046 4007 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.691470 4000 consensus_queue.cc:237] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:26.773556 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:26.774031 32428 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.731057 3379 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.778080 4007 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.784730 3811 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:26.743525 3582 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.792922 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.813776 32428 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.889374 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.890659 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.891283 32738 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:26.867218 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:26.871685 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:26.744983 3779 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.879117 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.897152 32471 raft_consensus.cc:1226] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 04:01:26.865559 32467 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:26.892033 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.908481 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.909317 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:26.906646 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 397399us
I20260806 04:01:26.907582 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:26.902086 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 88367.1us
I20260806 04:01:26.913216 32428 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.903963 32749 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:26.907573 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.909108 32748 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:26.772917 3870 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:26.917330 32428 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.856726 3905 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:26.929895 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.922160 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.919726 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.935600 32739 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:26.920928 32428 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:26.932327 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.929106 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.937304 32608 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 1.
I20260806 04:01:26.937660 32428 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:26.937935 32746 raft_consensus.cc:1226] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260806 04:01:26.873875 3811 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:26.919860 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 341518us
W20260806 04:01:26.933790 32553 connection.cc:232] RPC call timeout handler was delayed by -1.0095548301469535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1978s for a response
I20260806 04:01:26.747905 3926 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.942157 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:26.942934 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:26.915406 3582 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:26.916705 4043 consensus_meta.cc:287] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.521s user 0.005s sys 0.006s
I20260806 04:01:26.946637 32682 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:26.915632 3711 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:26.941586 3994 raft_consensus.cc:2781] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:26.918720 4041 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.931280 3984 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:26.958679 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:26.957883 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:27.039903 32467 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:26.959398 32741 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:26.944494 3811 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:26.918040 3434 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:26.950681 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.947814 3636 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:26.985872 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:26.986557 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:27.059698 1189 raft_consensus.cc:2379] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:26.987196 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.966531 4043 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:27.025419 3994 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:26.978434 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.949268 3905 raft_consensus.cc:2726] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:26.988211 32466 raft_consensus.cc:1261] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:26.980149 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.065629 32603 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:26.980837 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.066540 32604 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:26.981487 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.067989 32609 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:26.978770 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.979496 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.988785 3379 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.989420 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:26.989634 4066 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260806 04:01:26.917768 3367 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:26.994396 4006 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:26.995225 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:26.997740 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:26.999962 32746 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:27.002736 32427 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:27.186348 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 183690us
I20260806 04:01:27.188043 32427 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:27.005963 3725 raft_consensus.cc:3032] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:27.007040 3698 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.007440 32551 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:27.007818 3569 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:26.978411 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.196439 32615 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:26.976380 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:26.939121 3808 consensus_queue.cc:1046] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260806 04:01:26.967832 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.202625 32605 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:26.955267 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:26.962713 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.011364 4042 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:26.981374 3790 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.571s user 0.011s sys 0.000s
I20260806 04:01:27.057529 32477 raft_consensus.cc:1261] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:01:27.061472 3779 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:26.941740 3590 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:27.189956 3725 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:27.041924 3984 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:27.073915 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.099859 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.073561 4014 raft_consensus.cc:1050] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
I20260806 04:01:27.162317 32743 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:27.068382 3994 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:27.059250 3694 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.073009 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:27.044814 3261 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:27.180946 32745 raft_consensus.cc:1226] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:27.166282 1192 raft_consensus.cc:2379] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:27.086364 32610 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:27.184108 32470 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
W20260806 04:01:27.193146 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 185760us
I20260806 04:01:27.252527 32551 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:27.203859 4036 consensus_queue.cc:1046] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260806 04:01:27.198050 32427 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.181414 4006 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:27.100028 4043 consensus_queue.cc:1035] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20260806 04:01:27.190131 3711 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:27.235600 32478 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:27.059026 1187 raft_consensus.cc:2362] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5. Re-sending same reply.
I20260806 04:01:27.226882 1182 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:27.203752 3434 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:27.209833 3808 consensus_queue.cc:1225] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer ae0b3e43e7a64f789e51018fd5518d23 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:01:27.245369 32467 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:27.269930 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 71967.8us
I20260806 04:01:27.167256 3800 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.258143 32603 raft_consensus.cc:1261] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:01:27.229529 32602 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:27.241644 3984 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:27.193895 4018 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.203564 3698 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:27.263047 4080 log.cc:924] Time spent T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260806 04:01:27.279536 32427 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:27.264235 3997 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:27.244345 3779 consensus_queue.cc:1035] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260806 04:01:27.238655 3590 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:27.290398 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 340669us
W20260806 04:01:27.296404 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 350344us
W20260806 04:01:27.289323 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 345637us
I20260806 04:01:27.267253 3367 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:27.251706 3694 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.281109 3905 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:27.288547 4018 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:27.297194 1137 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:27.299089 32553 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
W20260806 04:01:27.286166 4062 log.cc:924] Time spent T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.618s user 0.000s sys 0.005s
I20260806 04:01:27.299571 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:27.283284 3434 raft_consensus.cc:686] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:27.322154 1137 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:27.307933 32553 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.265440 4006 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:27.326639 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 380054us
I20260806 04:01:27.324707 32553 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:27.328646 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 343436us
I20260806 04:01:27.296947 3997 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:27.327781 32682 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:27.329109 32427 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.335618 1136 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:27.309894 3367 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:27.304934 3905 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.320700 4007 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:27.366899 4007 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:27.315877 4018 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.344183 3698 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:27.343777 3434 consensus_queue.cc:237] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.344797 32739 raft_consensus.cc:3032] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:27.379093 32739 raft_consensus.cc:729] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:27.379904 32739 consensus_queue.cc:260] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:27.328920 3933 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:27.381117 3590 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:27.327111 4036 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:01:27.382190 4081 log.cc:924] Time spent T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20260806 04:01:27.284004 3725 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:27.364964 3800 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:27.365494 4072 log.cc:924] Time spent T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.357s user 0.002s sys 0.000s
I20260806 04:01:27.289737 3811 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.340694 3790 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:27.384898 3711 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:27.399591 3711 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.400012 3711 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:27.331727 3174 consensus_queue.cc:1046] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260806 04:01:27.404398 3174 raft_consensus.cc:3032] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:27.404768 3174 raft_consensus.cc:729] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:27.405583 3174 consensus_queue.cc:260] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:27.407409 3974 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.658s user 0.006s sys 0.003s
I20260806 04:01:27.407845 3974 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.409888 3974 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:27.387761 4008 consensus_meta.cc:287] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.599s user 0.005s sys 0.005s
I20260806 04:01:27.335573 3997 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:27.342759 4003 consensus_meta.cc:287] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.591s user 0.002s sys 0.009s
I20260806 04:01:27.360343 32427 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:27.349224 4043 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:27.433609 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 73357.7us
I20260806 04:01:27.411082 32474 raft_consensus.cc:1226] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:27.435905 1137 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:27.389040 4083 log.cc:924] Time spent T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
I20260806 04:01:27.387938 3855 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:27.387465 3870 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.428915 3811 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:27.410734 3779 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260806 04:01:27.388316 3367 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:27.429988 4008 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.365885 3905 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:27.395707 32682 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:27.502545 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 106909us
I20260806 04:01:27.502771 3782 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:27.406804 3174 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.437511 3855 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:27.504612 3855 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:27.437279 3790 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:27.437780 32467 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:27.509137 4008 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:27.509711 32683 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:27.510224 4018 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:27.510782 3994 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.510942 4071 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:27.511878 4071 raft_consensus.cc:686] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:27.513411 32477 raft_consensus.cc:1226] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 04:01:27.438367 32427 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.444489 1137 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:27.515112 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 70706.7us
W20260806 04:01:27.515252 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 76969.3us
I20260806 04:01:27.516049 1137 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:27.371834 3984 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.466455 4041 raft_consensus.cc:3032] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:27.481058 3793 raft_consensus.cc:2932] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:27.470041 3711 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:27.484647 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:27.491503 32739 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.493764 3725 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:27.494867 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1863) took 1152 ms. Trace:
I20260806 04:01:27.433789 4003 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.442643 4016 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20260806 04:01:27.442540 3870 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:27.519948 3854 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:27.516021 32427 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.534785 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:27.539645 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:27.547029 32470 raft_consensus.cc:1226] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:27.444849 3569 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:27.565586 3569 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.566061 3569 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:27.534329 4063 log.cc:924] Time spent T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.881s user 0.008s sys 0.000s
I20260806 04:01:27.522418 3790 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.512801 4071 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.547955 32469 rpcz_store.cc:276] 0806 04:01:26.342881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:26.423610 (+ 80729us) service_pool.cc:227] Handling call
0806 04:01:26.869804 (+446194us) spinlock_profiling.cc:243] Waited 7.83 ms on lock 0x7b5800003398. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:26.870023 (+ 219us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:26.940148 (+ 70125us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 04:01:26.954856 (+ 14708us) log.cc:842] Serialized 14 byte log entry
0806 04:01:27.293932 (+339076us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:27.293987 (+ 55us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:27.294028 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:27.494638 (+200610us) spinlock_profiling.cc:243] Waited 69.9 ms on lock 0x7b54002030d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:27.494683 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:27.494807 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15707,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":2714,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6344,"spinlock_wait_cycles":178804480,"thread_start_us":338704,"threads_started":1,"wal-append.queue_time_us":435883}
I20260806 04:01:27.591629 32746 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:01:27.583556 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:27.585562 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:27.587641 32748 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:27.593935 32326 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39766 (request call id 337) took 9556 ms. Trace:
I20260806 04:01:27.586748 3756 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:27.572338 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:27.613137 1192 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:27.614043 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:27.573794 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:27.616478 1189 raft_consensus.cc:2354] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8.
I20260806 04:01:27.617305 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:27.574198 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:27.576721 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 60798.2us
I20260806 04:01:27.618844 32427 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.526479 4041 raft_consensus.cc:729] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:27.456171 4092 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.146s
I20260806 04:01:27.549471 4018 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:27.528430 4087 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:27.624540 4087 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:27.624943 4087 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:27.628242 4097 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.612856 3582 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260806 04:01:27.628702 4097 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.629228 4097 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.582909 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:27.596283 32326 rpcz_store.cc:276] 0806 04:01:18.037611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:18.037766 (+ 155us) service_pool.cc:227] Handling call
0806 04:01:27.593840 (+9556074us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:23.814393 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:23.888915 (+ 74522us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:23.888950 (+ 35us) write_op.cc:435] Acquired schema lock
0806 04:01:23.888966 (+ 16us) tablet.cc:660] Decoding operations
0806 04:01:23.889599 (+ 633us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:23.889657 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:23.889685 (+ 28us) tablet.cc:683] Acquiring locks for 4 operations
0806 04:01:23.890065 (+ 380us) tablet.cc:699] Row locks acquired
0806 04:01:23.890084 (+ 19us) write_op.cc:260] PREPARE: finished
0806 04:01:23.890239 (+ 155us) write_op.cc:270] Start()
0806 04:01:23.890349 (+ 110us) write_op.cc:276] Timestamp: P: 1785988883890192 usec, L: 0
0806 04:01:23.890370 (+ 21us) op_driver.cc:331] REPLICATION: starting
0806 04:01:23.890990 (+ 620us) log.cc:842] Serialized 1703 byte log entry
0806 04:01:26.670132 (+2779142us) op_driver.cc:444] REPLICATION: finished
0806 04:01:27.076278 (+406146us) write_op.cc:301] APPLY: starting
0806 04:01:27.181178 (+104900us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:27.218101 (+ 36923us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:27.243935 (+ 25834us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:27.347713 (+103778us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:27.347925 (+ 212us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0806 04:01:27.347973 (+ 48us) write_op.cc:312] APPLY: finished
0806 04:01:27.425261 (+ 77288us) log.cc:842] Serialized 41 byte log entry
0806 04:01:27.425445 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:27.425546 (+ 101us) write_op.cc:454] Released schema lock
0806 04:01:27.425586 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:27.425887 (+ 301us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:27.425916 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":541621,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":361784,"apply.run_cpu_time_us":10654,"apply.run_wall_time_us":359350,"num_ops":4,"prepare.queue_time_us":1539417,"prepare.run_cpu_time_us":56969,"prepare.run_wall_time_us":2207923,"raft.queue_time_us":335682,"raft.run_cpu_time_us":18918,"raft.run_wall_time_us":326102,"replication_time_us":2778290,"thread_start_us":3489975,"threads_started":4,"wal-append.queue_time_us":1763612}]]}
I20260806 04:01:27.584995 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:27.632326 1188 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:27.584434 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:27.557072 3870 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.583348 4014 consensus_queue.cc:237] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.600116 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:27.569118 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 233563us
I20260806 04:01:27.528651 3836 consensus_queue.cc:1046] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260806 04:01:27.638942 1136 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.588274 3790 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.643011 1136 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.609191 32739 raft_consensus.cc:1261] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260806 04:01:27.578712 3800 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:27.582659 1183 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 6 LEARNER]: Advancing to term 7
I20260806 04:01:27.620924 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:27.622040 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:27.652767 1186 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 6
I20260806 04:01:27.596947 3694 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:27.617934 1189 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:27.612761 3756 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:27.581666 3711 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:27.603857 32682 connection.cc:232] RPC call timeout handler was delayed by -2.1981970549095422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
I20260806 04:01:27.659183 32682 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:27.644381 32739 pending_rounds.cc:76] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: Aborting all ops after (but not including) 2
I20260806 04:01:27.595191 4006 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:27.650930 1185 raft_consensus.cc:1226] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 04:01:27.652915 32427 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.660621 32739 pending_rounds.cc:97] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 04:01:27.660943 32739 raft_consensus.cc:2866] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:27.661594 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2164) took 1146 ms. Trace:
I20260806 04:01:27.661710 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:27.582057 3751 consensus_queue.cc:1046] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260806 04:01:27.599694 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:27.660307 1136 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:27.661973 32739 rpcz_store.cc:276] 0806 04:01:26.515606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:26.940295 (+424689us) service_pool.cc:227] Handling call
0806 04:01:26.940511 (+ 216us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:27.344641 (+404130us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b54002c3920. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d1e4730 00007fd16d256300 00007fd16d257371 00007fd16d253edf 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc
0806 04:01:27.491052 (+146411us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b58004af0b0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d1cff63 00007fd16d233aba 00007fd16d24b3af 00007fd16d24cb4e 00007fd16d256c89 00007fd16d257556 00007fd16d253edf 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684
0806 04:01:27.661369 (+170317us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:27.661545 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1020514944}
I20260806 04:01:27.663226 32427 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:27.667718 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 328893us
I20260806 04:01:27.648514 3790 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.668964 32682 leader_election.cc:398] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8.
I20260806 04:01:27.670872 32682 leader_election.cc:401] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 04:01:27.655565 3756 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:27.648504 3800 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.675869 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.680495 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.655921 3704 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:27.682431 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:27.684631 32603 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:27.685009 32427 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:27.667929 4018 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.664232 4016 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 04:01:27.676798 3995 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260806 04:01:27.624352 4041 consensus_queue.cc:260] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:27.689657 4018 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.706596 4018 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.686702 32682 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:27.665751 3694 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:27.708484 3694 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:27.708860 3694 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:27.658269 4043 consensus_queue.cc:1046] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260806 04:01:27.689440 32605 raft_consensus.cc:1226] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 04:01:27.689997 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.713097 32614 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:27.691768 32553 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:27.671072 4000 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.006s sys 0.015s
I20260806 04:01:27.694864 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.696200 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:27.686203 32610 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:27.683059 3800 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:27.721899 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:01:27.729889 4000 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s user 0.006s sys 0.016s
I20260806 04:01:27.735383 4008 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.735796 4008 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:27.736150 4008 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.736744 3997 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.731642 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.741340 3434 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:27.741744 3434 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:27.742116 3434 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:27.703307 3984 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:27.728096 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.745266 32602 raft_consensus.cc:2379] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:27.745363 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.745698 3855 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.693871 3751 raft_consensus.cc:3032] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:27.681027 3636 consensus_queue.cc:1035] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260806 04:01:27.728941 3704 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:27.714453 1186 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
W20260806 04:01:27.752701 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 66080.3us
I20260806 04:01:27.764125 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:27.708757 4003 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:27.756173 3994 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:27.756282 4043 raft_consensus.cc:2726] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:27.756857 4000 raft_consensus.cc:3032] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:27.715296 4012 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.748570 3997 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:27.811228 32682 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:27.812299 32682 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:27.763062 3704 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:27.765919 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 244191us
I20260806 04:01:27.766464 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:27.768551 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.778050 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:27.823163 32749 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:27.691233 3905 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:27.761562 3751 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:27.815724 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:27.827883 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 59417us
I20260806 04:01:27.828706 32553 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.829869 32553 leader_election.cc:302] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.764745 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1697) took 1001 ms. Trace:
I20260806 04:01:27.834087 32553 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.835649 32682 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:27.836191 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.838443 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.839294 32553 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.840161 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:27.840955 32553 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:27.841948 3926 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.842415 3926 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:27.842837 3926 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.853909 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:27.854947 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:27.856127 32610 raft_consensus.cc:1226] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:27.856288 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:27.857157 32470 raft_consensus.cc:2379] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:27.857928 32610 raft_consensus.cc:3032] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:27.858343 32610 raft_consensus.cc:729] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:27.859014 32610 consensus_queue.cc:260] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:27.859397 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:27.849084 4102 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:27.772277 4085 consensus_queue.cc:1035] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.089s
I20260806 04:01:27.859890 32610 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.862494 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:27.863911 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:27.864686 32466 raft_consensus.cc:2379] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:27.865443 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:27.796000 4042 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:27.866184 32475 raft_consensus.cc:2354] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:27.879103 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:27.851770 1137 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:27.795897 4000 raft_consensus.cc:729] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:27.855258 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:27.860858 32469 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:01:27.796154 4012 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:27.866515 32474 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:27.880259 32746 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:27.827157 3704 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:27.848582 3590 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:27.825438 3782 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8.
W20260806 04:01:27.935987 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 70381.2us
I20260806 04:01:27.860816 3791 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:27.851560 3933 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:27.851397 3379 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:27.848352 3711 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:28.020701 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:27.948904 32682 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:27.924188 4012 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:27.849326 4007 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:27.936849 3647 consensus_queue.cc:1046] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:27.877993 4043 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:28.037133 4043 raft_consensus.cc:3032] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:28.037487 4043 raft_consensus.cc:729] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:28.037737 4091 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:28.038277 4043 consensus_queue.cc:260] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:27.869046 4095 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:28.039826 4095 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:28.040211 4095 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:27.922706 3779 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:27.848610 4098 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:28.002473 3994 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:28.038811 4092 raft_consensus.cc:3032] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:27.850646 4089 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:27.953715 3791 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:28.063426 3779 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:28.002415 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:27.832585 32741 rpcz_store.cc:276] 0806 04:01:26.763611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.074674 (+311063us) service_pool.cc:227] Handling call
0806 04:01:27.156688 (+ 82014us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:27.156917 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:27.434292 (+277375us) log.cc:842] Serialized 14 byte log entry
0806 04:01:27.764310 (+330018us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:27.764363 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:27.764408 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:27.764467 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:27.764691 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":430083,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":78,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":11968,"spinlock_wait_cycles":16128,"thread_start_us":583039,"threads_started":2,"wal-append.queue_time_us":422139}
W20260806 04:01:28.104494 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 155697us
I20260806 04:01:28.105723 32682 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:28.108374 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 256662us
I20260806 04:01:28.112447 32749 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:28.124500 32682 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:28.081223 4092 raft_consensus.cc:729] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:27.931985 3751 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.092393 3791 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:28.128813 1137 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:28.107945 3779 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:28.100433 4042 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:28.151221 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:28.086239 3994 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.128262 4105 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:28.145846 4043 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:28.162546 32610 raft_consensus.cc:1261] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260806 04:01:28.188181 32610 pending_rounds.cc:76] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 2
I20260806 04:01:28.188582 32610 pending_rounds.cc:97] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 04:01:28.188915 32610 raft_consensus.cc:2866] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 04:01:28.169503 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 484531us
I20260806 04:01:28.131260 32740 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:28.181175 1136 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.050s sys 0.070s
I20260806 04:01:28.145455 3811 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:28.193905 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 352952us
W20260806 04:01:28.202281 1136 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.537s user 0.050s sys 0.070s
W20260806 04:01:28.206459 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 546171us
I20260806 04:01:28.207474 32610 raft_consensus.cc:1226] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 04:01:28.082206 4000 consensus_queue.cc:260] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.214570 32427 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:28.240844 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 116448us
I20260806 04:01:28.161554 4003 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:01:28.246910 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 95724.5us
I20260806 04:01:28.249079 1137 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:28.255803 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 44615.5us
I20260806 04:01:28.272552 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:28.262902 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:28.201435 4042 raft_consensus.cc:2741] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 04:01:28.274176 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:28.205518 3905 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.281802 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:28.285166 32478 raft_consensus.cc:1226] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:28.202392 3855 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:28.211509 4105 raft_consensus.cc:686] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:28.239214 3697 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:28.220172 4098 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:28.275648 4066 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20260806 04:01:28.302737 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:28.247453 3698 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:28.275331 4121 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:28.308007 4127 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:28.275065 4119 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 04:01:28.325505 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 76501.2us
I20260806 04:01:28.219161 3811 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:28.428750 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 187056us
I20260806 04:01:28.432246 32682 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:28.315352 4110 log.cc:924] Time spent T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.313s user 0.001s sys 0.000s
I20260806 04:01:28.294385 3697 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:28.388780 4137 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.016s
W20260806 04:01:28.381829 4112 log.cc:924] Time spent T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.230s user 0.000s sys 0.000s
W20260806 04:01:28.382582 4111 log.cc:924] Time spent T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.301s user 0.000s sys 0.001s
W20260806 04:01:28.380915 4125 log.cc:924] Time spent T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260806 04:01:28.395545 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:28.275828 3997 consensus_queue.cc:1046] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260806 04:01:28.289059 3379 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:28.324677 3756 consensus_meta.cc:287] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.635s user 0.014s sys 0.005s
I20260806 04:01:28.471345 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:28.288998 4089 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:28.365216 32553 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:28.221483 4092 consensus_queue.cc:260] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.424005 1137 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:28.383989 4072 log.cc:924] Time spent T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
W20260806 04:01:28.438791 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 103825us
W20260806 04:01:28.419297 3926 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.576s user 0.011s sys 0.007s
W20260806 04:01:28.417623 4109 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.333s user 0.001s sys 0.000s
I20260806 04:01:28.368602 4135 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:28.380959 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.457638 1186 raft_consensus.cc:2354] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4.
I20260806 04:01:28.444994 3697 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:28.447523 4087 consensus_meta.cc:287] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.822s user 0.013s sys 0.001s
W20260806 04:01:28.444329 3790 consensus_meta.cc:287] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.762s user 0.011s sys 0.014s
W20260806 04:01:28.452245 3870 consensus_meta.cc:287] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.819s user 0.004s sys 0.010s
W20260806 04:01:28.452375 3569 consensus_meta.cc:287] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.881s user 0.012s sys 0.008s
I20260806 04:01:28.421908 4119 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:28.456836 3694 consensus_meta.cc:287] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.747s user 0.009s sys 0.001s
I20260806 04:01:28.322453 4098 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:28.489938 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 124794us
I20260806 04:01:28.487938 1182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:45672
W20260806 04:01:28.490820 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 66985.5us
W20260806 04:01:28.472477 4128 log.cc:924] Time spent T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260806 04:01:28.440109 4105 consensus_queue.cc:237] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.435640 4127 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.520825 3790 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.626688 4136 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
W20260806 04:01:28.529305 4018 consensus_meta.cc:287] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.822s user 0.010s sys 0.000s
I20260806 04:01:28.577967 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1699) took 1633 ms. Trace:
I20260806 04:01:28.567080 32551 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:28.628417 1137 leader_election.cc:398] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:28.503933 32471 raft_consensus.cc:1226] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 04:01:28.503430 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1880) took 1116 ms. Trace:
W20260806 04:01:28.611727 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 179571us
I20260806 04:01:28.636300 1137 leader_election.cc:401] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:28.638942 32682 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:28.499895 4097 consensus_meta.cc:287] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.870s user 0.010s sys 0.000s
W20260806 04:01:28.499976 4071 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.871s user 0.002s sys 0.037s
W20260806 04:01:28.542737 3854 consensus_meta.cc:287] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.965s user 0.003s sys 0.015s
W20260806 04:01:28.611402 3704 consensus_meta.cc:287] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.657s user 0.011s sys 0.001s
W20260806 04:01:28.534947 3725 consensus_meta.cc:287] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.978s user 0.012s sys 0.008s
I20260806 04:01:28.638454 32473 rpcz_store.cc:276] 0806 04:01:27.386735 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.386908 (+ 173us) service_pool.cc:227] Handling call
0806 04:01:27.399054 (+ 12146us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:27.411724 (+ 12670us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:27.411754 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:27.412220 (+ 466us) log.cc:842] Serialized 26 byte log entry
0806 04:01:27.685783 (+273563us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:27.685822 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:27.687676 (+ 1854us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:28.503181 (+815505us) raft_consensus.cc:1671] finished
0806 04:01:28.503204 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:28.503364 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22128,"raft.queue_time_us":16528,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":236060,"threads_started":1,"wal-append.queue_time_us":400966}
W20260806 04:01:28.521536 4008 consensus_meta.cc:287] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.785s user 0.004s sys 0.004s
W20260806 04:01:28.578660 4132 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20260806 04:01:28.654724 32467 raft_consensus.cc:1226] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:28.495378 4006 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:28.519797 4041 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:28.578912 3261 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:28.482921 3379 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.535130 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33964 (request call id 792) took 1558 ms. Trace:
I20260806 04:01:28.634374 4119 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:28.659476 32610 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260806 04:01:28.657022 3434 consensus_meta.cc:287] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.914s user 0.009s sys 0.002s
W20260806 04:01:28.645864 4071 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.017s user 0.004s sys 0.046s
I20260806 04:01:28.635149 32745 rpcz_store.cc:276] 0806 04:01:26.944579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.251927 (+307348us) service_pool.cc:227] Handling call
0806 04:01:27.286599 (+ 34672us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:27.300351 (+ 13752us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:27.300381 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:27.483645 (+183264us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b2c00108868. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:27.484482 (+ 837us) log.cc:842] Serialized 598 byte log entry
0806 04:01:27.818456 (+333974us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:27.818494 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:27.818556 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:28.577679 (+759123us) raft_consensus.cc:1671] finished
0806 04:01:28.577698 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:28.577904 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3561,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":6947,"spinlock_wait_cycles":10039552,"thread_start_us":333591,"threads_started":1,"wal-append.queue_time_us":603029}
I20260806 04:01:28.553088 3997 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:28.540638 4164 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:28.527562 3984 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:28.519946 3811 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:28.495779 3994 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.659608 32682 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:28.647567 1182 raft_consensus.cc:595] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:28.499691 3926 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.657642 3854 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:28.683163 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:28.683461 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:28.656139 4151 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
W20260806 04:01:28.673262 1183 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.017s user 0.006s sys 0.014s
I20260806 04:01:28.628340 3569 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.634835 3870 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.684890 3997 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.694835 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:28.680795 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 311795us
I20260806 04:01:28.544461 4087 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.700114 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:28.641566 4018 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.637890 3694 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.714095 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:28.715602 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:28.706159 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:28.673246 3725 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:28.489763 4089 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:28.678995 4012 consensus_meta.cc:287] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.608s user 0.004s sys 0.005s
I20260806 04:01:28.712886 1185 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:28.673599 3590 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:28.671461 32612 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:28.671757 4139 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:28.722976 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:28.723632 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:28.724460 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:28.724790 1190 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:01:28.725327 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:28.704917 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 78966.8us
I20260806 04:01:28.719233 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:28.719502 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:28.722198 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:28.692876 1182 raft_consensus.cc:603] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:28.517601 4113 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:28.658869 3704 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.692724 3811 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:28.729476 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:28.727128 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 160150us
I20260806 04:01:28.692624 3647 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:28.653167 4097 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.666097 4008 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.480170 3756 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.737327 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:28.739315 1190 raft_consensus.cc:1226] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:28.740056 1185 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:28.671269 32738 rpcz_store.cc:276] 0806 04:01:26.976696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.259202 (+282506us) service_pool.cc:227] Handling call
0806 04:01:28.534172 (+1274970us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54000a87d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d25cf2a 00007fd172199c6e 00007fd16b890807 00007fd16b890792 00007fd16b8906c1 00007fd16b89063c 00007fd16b88f8f2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:28.535029 (+ 857us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1944784256}
I20260806 04:01:28.722599 3725 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.692760 3793 consensus_queue.cc:1046] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.054s
I20260806 04:01:28.751086 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.775595 1185 raft_consensus.cc:1226] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:28.765451 32749 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:28.756253 4162 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:28.777135 1185 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 2 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:01:28.777328 32551 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:28.741232 1183 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 7 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:28.678661 4144 consensus_queue.cc:1046] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260806 04:01:28.762583 3704 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:28.762310 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.715009 1184 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:28.746831 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:28.691636 3984 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.808130 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1841) took 1022 ms. Trace:
I20260806 04:01:28.808563 32604 rpcz_store.cc:276] 0806 04:01:27.785374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.821603 (+ 36229us) service_pool.cc:227] Handling call
0806 04:01:27.856921 (+ 35318us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:28.091604 (+234683us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:28.091636 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:28.091675 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:28.091702 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:28.807896 (+716194us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400087d54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:28.807936 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:28.808055 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":539211520}
I20260806 04:01:28.734452 4043 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:28.780273 1189 raft_consensus.cc:1226] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260806 04:01:28.810129 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 69144.2us
I20260806 04:01:28.781293 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.734372 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1832) took 1043 ms. Trace:
I20260806 04:01:28.781351 32609 raft_consensus.cc:2354] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4.
I20260806 04:01:28.781904 32469 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:28.784325 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:28.676429 3434 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.757251 3811 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.776016 3790 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.685070 4164 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.871124 3926 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:28.729995 3590 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:28.850807 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 79881.6us
I20260806 04:01:28.851495 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.764050 4152 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:28.800766 4167 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.280s user 0.003s sys 0.000s
I20260806 04:01:28.857493 1188 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:28.722457 4012 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.877382 4089 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.865603 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:28.787629 4018 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.870854 32614 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:28.775398 4003 consensus_queue.cc:1035] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260806 04:01:28.873562 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.979194 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.979769 32604 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:28.979914 3779 consensus_meta.cc:287] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.804s user 0.008s sys 0.002s
I20260806 04:01:28.980396 3779 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.981489 32745 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:28.982288 4089 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:28.982854 3697 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.877357 3434 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.877910 4172 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:28.985001 32475 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:28.877820 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:28.878199 4097 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:28.986111 4095 consensus_meta.cc:287] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.918s user 0.010s sys 0.000s
I20260806 04:01:28.985185 4172 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.808037 4162 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.880324 3984 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:28.833806 4071 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:28.852039 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 693) took 1278 ms. Trace:
I20260806 04:01:28.885752 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:28.850663 32602 raft_consensus.cc:1226] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:28.924062 32749 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:01:28.924244 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:28.925951 3569 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.747437 4113 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:28.882148 32740 consensus_meta.cc:287] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.633s user 0.007s sys 0.003s
I20260806 04:01:28.764453 4068 consensus_queue.cc:1046] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.119s
I20260806 04:01:28.864637 32612 rpcz_store.cc:276] 0806 04:01:27.690797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.692622 (+ 1825us) service_pool.cc:227] Handling call
0806 04:01:28.380774 (+688152us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b58001e5798. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:28.729501 (+348727us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24442752}
I20260806 04:01:28.881209 3694 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.004448 3811 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:28.950193 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:29.011909 4172 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:28.999490 4097 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.033373 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:28.962767 4008 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.033619 4097 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:28.963083 4007 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.024150 32740 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:29.034631 4007 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.035331 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33182 (request call id 447) took 1183 ms. Trace:
I20260806 04:01:28.964525 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
W20260806 04:01:28.999672 3800 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 1.246s user 0.019s sys 0.003s
W20260806 04:01:29.036412 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 71938.4us
I20260806 04:01:29.036316 3800 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:29.037119 4007 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.037458 32477 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:29.037577 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.033767 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.966225 1190 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:28.785272 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.039228 4119 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.039899 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.039651 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:28.969650 3870 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.969429 3590 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:28.879241 4087 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:28.944657 3791 consensus_meta.cc:287] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.790s user 0.010s sys 0.000s
I20260806 04:01:29.041095 32605 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:29.041308 3791 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:28.999475 4095 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.042488 32608 raft_consensus.cc:1226] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:29.044914 3791 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.008924 3697 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.049515 3905 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:01:29.004240 3997 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:28.967861 4152 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.032565 3434 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:28.964620 4012 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:28.763949 4189 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:29.046003 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.046450 4191 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:29.015638 4162 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.788364 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:29.047679 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.048347 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:29.040493 32610 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:29.069958 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.072325 1188 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.073349 32603 raft_consensus.cc:2379] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 7: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:29.048952 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.035843 32740 rpcz_store.cc:276] 0806 04:01:27.852386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.867657 (+ 15271us) service_pool.cc:227] Handling call
0806 04:01:29.035270 (+1167613us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:29.077471 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.078209 4164 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.024507 1183 rpcz_store.cc:276] 0806 04:01:27.573142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.577633 (+ 4491us) service_pool.cc:227] Handling call
0806 04:01:27.582523 (+ 4890us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:28.782352 (+1199829us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:28.836305 (+ 53953us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:29.004448 3756 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.003775 4071 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.024825 32749 raft_consensus.cc:595] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:29.080931 32749 raft_consensus.cc:603] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:29.034243 32612 raft_consensus.cc:2379] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:29.046391 4136 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:29.105726 4141 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.035589 4097 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.069350 3434 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.056622 4113 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:28.961752 4127 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.081077 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.124943 32741 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.057536 1190 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:29.089033 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.118700 32427 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.126264 32553 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
W20260806 04:01:29.095052 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 316883us
I20260806 04:01:29.097059 32610 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:29.130443 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.060930 3800 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.101042 3854 raft_consensus.cc:3032] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:29.123207 32603 raft_consensus.cc:1226] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:29.030745 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.080701 4152 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:29.109824 4141 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.062788 4068 raft_consensus.cc:3032] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:29.116837 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1886) took 1530 ms. Trace:
I20260806 04:01:29.146572 32467 rpcz_store.cc:276] 0806 04:01:27.585937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.586176 (+ 239us) service_pool.cc:227] Handling call
0806 04:01:27.586491 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:28.658310 (+1071819us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:29.116596 (+458286us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540009cc54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:29.116774 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2456308608}
I20260806 04:01:29.113430 1187 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.047725 3811 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:28.789505 32682 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:29.150035 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 360597us
I20260806 04:01:29.069729 3379 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.101955 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:01:29.136480 32428 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:29.136991 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:29.121598 32604 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.142778 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:29.147579 32605 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:29.118373 3779 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.152448 32478 raft_consensus.cc:1226] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
W20260806 04:01:29.247875 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 470583us
I20260806 04:01:29.248834 32551 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:29.206029 32615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.000s sys 0.016s
W20260806 04:01:29.249727 32615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s user 0.003s sys 0.017s
I20260806 04:01:29.133219 3697 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:29.254504 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 111777us
W20260806 04:01:29.256029 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 120041us
I20260806 04:01:29.258172 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:29.255587 32605 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 04:01:29.258976 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:01:29.255728 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 119319us
I20260806 04:01:29.262547 32428 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.260987 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.263576 32748 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:29.266101 32551 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.252892 32612 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:29.275665 32428 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.276109 4192 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:29.276829 4192 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:29.278033 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.146492 4068 raft_consensus.cc:729] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:29.246068 3933 raft_consensus.cc:3032] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:29.293532 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.271221 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.214843 4071 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:29.280350 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 122119us
I20260806 04:01:29.277246 3926 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:29.276753 3997 consensus_queue.cc:1046] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.005s
I20260806 04:01:29.281229 3590 consensus_queue.cc:1046] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:29.164170 4152 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:29.277350 4192 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:29.282905 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.283198 32603 raft_consensus.cc:1226] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:29.132848 4085 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.269740 3379 consensus_queue.cc:1046] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260806 04:01:29.281666 3870 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 04:01:29.328306 4068 consensus_queue.cc:260] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.330060 4068 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:29.326575 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.313535 4119 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.332155 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.322641 4197 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:29.327411 4085 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:29.369849 4202 log.cc:924] Time spent T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260806 04:01:29.368023 4197 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:29.374461 4197 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:29.374667 3751 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.376564 4085 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:29.378530 4180 log.cc:924] Time spent T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20260806 04:01:29.384058 4000 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:29.376703 4141 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:29.389714 4089 raft_consensus.cc:3032] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:29.390132 4089 raft_consensus.cc:729] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:29.391366 4089 consensus_queue.cc:260] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.391943 4192 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.395735 4192 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.396298 4098 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.401345 4189 raft_consensus.cc:3032] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:29.401832 4189 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:29.284000 32467 raft_consensus.cc:1226] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
W20260806 04:01:29.446655 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 146309us
I20260806 04:01:29.449292 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:29.450637 32467 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:29.284116 32470 raft_consensus.cc:1226] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:29.284324 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.287909 32553 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:29.463419 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.477149 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1888) took 1547 ms. Trace:
W20260806 04:01:29.477424 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 189510us
I20260806 04:01:29.479991 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.480855 32602 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.464306 32322 catalog_manager.cc:5495] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:29.467110 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:29.484746 32467 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:29.485169 32614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1836) took 1769 ms. Trace:
I20260806 04:01:29.477401 32603 raft_consensus.cc:2379] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:29.485890 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.486605 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:29.487362 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:29.488059 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.472944 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1828) took 1792 ms. Trace:
I20260806 04:01:29.475998 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:29.279765 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 129145us
I20260806 04:01:29.319486 32738 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 04:01:29.299616 32748 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:29.339295 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.320029 1137 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:29.220393 3261 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.552s user 0.011s sys 0.002s
I20260806 04:01:29.546805 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:29.530856 4098 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:29.157437 3836 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:29.551126 4098 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.551594 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.552646 32466 raft_consensus.cc:2379] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:29.552120 3836 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.132975 3854 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:29.553614 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.554423 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.554445 4098 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.555286 32610 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:29.293262 3933 raft_consensus.cc:729] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:29.157629 4041 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:29.559216 4041 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.561146 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.277868 4095 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.282387 4097 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:29.265895 4164 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:29.563164 4164 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:29.563661 4164 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:29.532464 4192 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:29.564240 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1843) took 1756 ms. Trace:
I20260806 04:01:29.275457 4190 consensus_queue.cc:1046] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260806 04:01:29.564711 32608 rpcz_store.cc:276] 0806 04:01:27.785830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.832121 (+ 46291us) service_pool.cc:227] Handling call
0806 04:01:27.837359 (+ 5238us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:29.042866 (+1205507us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:29.488737 (+445871us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400216e54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:29.542628 (+ 53891us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2771140352}
I20260806 04:01:29.292723 4191 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:29.281416 4193 raft_consensus.cc:2726] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 04:01:29.566530 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 246436us
I20260806 04:01:29.566880 4156 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:29.567553 4156 raft_consensus.cc:686] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:29.568856 4156 consensus_queue.cc:237] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.569806 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.313647 4200 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 04:01:29.572451 1184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 698) took 1988 ms (client timeout 1717 ms). Trace:
W20260806 04:01:29.572988 1184 rpcz_store.cc:269] 0806 04:01:27.583864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.585312 (+ 1448us) service_pool.cc:227] Handling call
0806 04:01:29.572007 (+1986695us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54004853d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:29.572234 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2528823552}
I20260806 04:01:29.574329 4190 raft_consensus.cc:3032] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:29.574978 4190 raft_consensus.cc:729] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:29.566210 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.577663 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.514931 32467 raft_consensus.cc:2379] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:29.514248 32737 raft_consensus.cc:2379] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:29.519423 3379 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.402819 4189 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 7.10, Last appended by leader: 10, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.521708 32478 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:29.521231 4197 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.528169 32682 leader_election.cc:398] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4.
I20260806 04:01:29.528895 3790 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.588658 3790 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.589349 32427 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:29.589742 4197 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.590611 4190 consensus_queue.cc:260] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.591583 3790 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.529800 4006 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:29.592603 4190 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.593108 3811 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.593969 4192 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:29.596829 32478 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:29.592597 4006 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:29.597610 4006 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.598294 3926 raft_consensus.cc:3032] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:29.593719 3811 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.598769 3926 raft_consensus.cc:729] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:29.598948 32470 raft_consensus.cc:2354] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5.
I20260806 04:01:29.600950 3791 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:29.601840 3791 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.531580 3711 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.602517 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1890) took 1609 ms. Trace:
I20260806 04:01:29.530984 4018 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.603799 4018 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.599882 3926 consensus_queue.cc:260] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.605638 3926 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:29.606227 4018 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.532569 3697 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.529971 3984 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.477556 32469 rpcz_store.cc:276] 0806 04:01:27.929105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.930858 (+ 1753us) service_pool.cc:227] Handling call
0806 04:01:27.984889 (+ 54031us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:28.865136 (+880247us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:29.461667 (+596531us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b5400260554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:29.476905 (+ 15238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1832556288}
I20260806 04:01:29.531891 4085 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.529031 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.541668 4178 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.610390 3984 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.610582 4085 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.611159 4189 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:29.533274 4203 consensus_queue.cc:1046] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:29.610960 4178 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.546279 4091 consensus_queue.cc:1046] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:29.547533 3261 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:29.317389 3698 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:29.549340 4007 consensus_queue.cc:1046] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:29.549541 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.550197 32738 pending_rounds.cc:76] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Aborting all ops after (but not including) 5
I20260806 04:01:29.630164 32738 pending_rounds.cc:97] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 04:01:29.630540 32738 raft_consensus.cc:2866] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:29.554944 3854 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.631693 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.632398 3698 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.632938 3854 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:29.586861 32682 leader_election.cc:401] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 04:01:29.538765 32605 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:29.597834 3379 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.602090 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:29.485528 32614 rpcz_store.cc:276] 0806 04:01:27.715255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.778721 (+ 63466us) service_pool.cc:227] Handling call
0806 04:01:27.821662 (+ 42941us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:28.978140 (+1156478us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:29.484740 (+506600us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400086954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:29.485007 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2412513152}
I20260806 04:01:29.488499 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1848) took 1538 ms. Trace:
I20260806 04:01:29.585668 32603 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.484949 32322 catalog_manager.cc:5495] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 3, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 3 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:29.593834 3434 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.617343 4091 raft_consensus.cc:3032] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:29.602967 32477 rpcz_store.cc:276] 0806 04:01:27.992711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.992839 (+ 128us) service_pool.cc:227] Handling call
0806 04:01:28.001162 (+ 8323us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:29.037860 (+1036698us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:29.601996 (+564136us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400012f54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:29.602291 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2382856064}
I20260806 04:01:29.593743 4041 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.640554 4193 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:29.641158 4193 raft_consensus.cc:686] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:29.642040 3379 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.628557 32467 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:29.642421 4193 consensus_queue.cc:237] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.643543 32612 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:29.643790 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 116091us
W20260806 04:01:29.644471 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 95082.7us
I20260806 04:01:29.644922 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1614) took 1364 ms. Trace:
I20260806 04:01:29.645284 32475 rpcz_store.cc:276] 0806 04:01:28.280109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:28.285578 (+ 5469us) service_pool.cc:227] Handling call
0806 04:01:28.285846 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:28.985388 (+699542us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:29.644501 (+659113us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b54001e05d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:29.644753 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1607469824}
I20260806 04:01:29.647552 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.644037 3647 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4.
I20260806 04:01:29.578915 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:29.627432 32466 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:29.515069 32615 rpcz_store.cc:276] 0806 04:01:27.680419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.686099 (+ 5680us) service_pool.cc:227] Handling call
0806 04:01:28.439321 (+753222us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b58001e5798. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:28.439574 (+ 253us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:28.440256 (+ 682us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:28.440288 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:28.453087 (+ 12799us) log.cc:842] Serialized 26 byte log entry
0806 04:01:29.250182 (+797095us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:29.250228 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:29.250296 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:29.472339 (+222043us) raft_consensus.cc:1671] finished
0806 04:01:29.472376 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:29.472754 (+ 378us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19866,"raft.queue_time_us":9782,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":13690,"spinlock_wait_cycles":33082752,"thread_start_us":759896,"threads_started":1,"wal-append.queue_time_us":879740}
W20260806 04:01:29.635588 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 46568.6us
I20260806 04:01:29.640065 4007 raft_consensus.cc:3032] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:29.652020 3811 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.653051 4095 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:29.655435 4098 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:29.649839 4091 raft_consensus.cc:729] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:29.656493 32608 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.651791 4003 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:01:29.651702 3697 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.653239 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:29.638298 32609 rpcz_store.cc:276] 0806 04:01:27.949662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:28.020404 (+ 70742us) service_pool.cc:227] Handling call
0806 04:01:29.487998 (+1467594us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b5400306454. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:29.488290 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1747271936}
I20260806 04:01:29.663443 4016 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:29.603135 3711 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.665202 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.666966 3711 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.642477 3725 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.668139 3836 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.669937 32746 raft_consensus.cc:1226] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:29.670488 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.671535 32740 raft_consensus.cc:2379] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:29.673357 4016 raft_consensus.cc:686] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:29.658252 4007 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:29.674527 4016 consensus_queue.cc:237] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.675129 1186 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.633780 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.678432 4092 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:29.657060 4091 consensus_queue.cc:260] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.685566 4091 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.686373 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.686866 4089 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:29.687270 32743 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:29.690447 32610 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:29.642271 32322 catalog_manager.cc:5495] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:29.696579 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.696738 4085 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.698441 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.699337 32746 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:29.703562 32737 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:29.704447 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.705557 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.706457 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.687497 3933 consensus_queue.cc:260] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.708003 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:29.708894 1188 raft_consensus.cc:2379] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:29.708956 3933 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.696548 32322 catalog_manager.cc:5495] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 3, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 3 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:29.697582 32614 raft_consensus.cc:1226] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:29.705897 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.716169 4200 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:29.718868 32322 catalog_manager.cc:5495] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:29.725832 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.726590 32477 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:29.724531 32428 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:29.710223 4003 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.716593 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:29.731341 1187 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:29.732259 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 824) took 1049 ms. Trace:
I20260806 04:01:29.675628 4007 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.729287 4192 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.733572 4007 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:29.733971 1192 raft_consensus.cc:1226] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:29.734750 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.735481 32474 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:29.735674 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.732551 1186 rpcz_store.cc:276] 0806 04:01:28.683119 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:28.683235 (+ 116us) service_pool.cc:227] Handling call
0806 04:01:29.721674 (+1038439us) spinlock_profiling.cc:243] Waited 925 ms on lock 0x7b54006342d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:29.732200 (+ 10526us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2126810368}
I20260806 04:01:29.737150 32475 raft_consensus.cc:1226] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:29.742821 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.720239 4085 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:29.716912 4200 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.747594 4124 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:29.748008 4124 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:29.748374 4124 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:29.731069 3984 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.746302 1185 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.761269 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.749455 32322 catalog_manager.cc:5495] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 4, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 4 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:29.773633 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:29.759784 32427 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.778107 32428 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:29.755688 4200 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.779831 32478 raft_consensus.cc:2379] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:29.785320 32428 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:29.786319 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.771519 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.786594 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.787392 32322 catalog_manager.cc:5495] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 3, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 3 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:29.794659 1185 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:29.789184 3791 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.797124 3811 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:29.797452 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:29.798131 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 48980.1us
I20260806 04:01:29.799122 32477 raft_consensus.cc:1226] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:29.783313 4085 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.756618 4003 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.800295 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:29.789152 4016 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:29.791612 4018 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:01:29.820811 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 130089us
I20260806 04:01:29.813603 32427 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.816982 1187 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:29.819039 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.820255 4018 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.822561 32604 raft_consensus.cc:1226] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:29.823060 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.802985 32322 catalog_manager.cc:5495] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:29.831940 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.835373 4006 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.820556 4192 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260806 04:01:29.826047 4018 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:29.820355 3990 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.830596 3697 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.829993 4016 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.860819 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.862982 32737 raft_consensus.cc:2379] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:29.866497 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.845534 32322 catalog_manager.cc:5495] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 3, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:29.870467 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.874120 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:29.875401 4233 log.cc:924] Time spent T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260806 04:01:29.846825 3984 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:29.877367 32741 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:29.878854 3990 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.885521 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.853718 4006 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.872953 4016 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:29.891084 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 91214.5us
I20260806 04:01:29.893438 32746 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:29.846995 3791 consensus_queue.cc:1046] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20260806 04:01:29.902945 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.896759 32322 catalog_manager.cc:5495] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 4, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 4 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:29.907622 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:29.911738 32746 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:29.895808 3984 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.915263 32553 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.895931 32602 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.896075 4200 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.924988 32427 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:29.900915 32612 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:29.907871 4178 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.926671 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.912279 3811 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:29.910844 4085 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.930202 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.932511 3926 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:29.934645 4003 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.934787 3647 consensus_queue.cc:1046] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:01:29.957330 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:29.957798 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.935870 4068 consensus_queue.cc:1046] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:29.936393 4139 consensus_queue.cc:1046] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:01:29.937515 4095 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:29.941984 1188 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:29.950896 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.951428 1137 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:29.951810 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.954061 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.955075 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.958436 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:29.963182 32474 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:29.945794 4098 raft_consensus.cc:3032] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:29.943167 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:29.965157 32745 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:29.967948 32738 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:29.969090 3791 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.969697 3791 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.971725 3791 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:29.957715 32467 raft_consensus.cc:2379] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 2: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:29.935444 4185 consensus_queue.cc:1046] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:01:29.963774 4098 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:29.962626 32322 catalog_manager.cc:5495] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:29.958086 4003 raft_consensus.cc:3032] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:29.983487 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:29.984043 4068 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Stepping down as leader of term 7
W20260806 04:01:29.987975 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 77802us
I20260806 04:01:29.986687 32603 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:29.989037 4008 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:29.989557 4008 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.990043 4105 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:29.992146 3990 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:29.992344 4068 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:29.992376 3647 raft_consensus.cc:3032] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:29.992483 4185 raft_consensus.cc:3032] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:29.985586 4003 raft_consensus.cc:729] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:29.982021 3984 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.987323 4200 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:29.995923 4012 consensus_queue.cc:1046] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:29.996559 4008 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:30.011631 3984 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.013302 4124 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:29.996809 1137 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:29.998586 32322 catalog_manager.cc:5495] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 1 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 3 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:29.972679 3697 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 04:01:30.008942 4237 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20260806 04:01:30.002913 4185 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:29.999379 3647 raft_consensus.cc:729] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:30.010643 32604 raft_consensus.cc:1226] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:30.009047 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:30.017022 1136 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:29.940061 4164 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.019992 32603 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 04:01:30.017957 3751 consensus_meta.cc:287] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.528s user 0.009s sys 0.005s
I20260806 04:01:30.024381 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:30.023854 32322 catalog_manager.cc:5495] T 42a553c7b71d493284e5e79b6f19f848 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:30.028774 32477 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:30.032222 32469 raft_consensus.cc:2354] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5.
I20260806 04:01:30.037988 1188 raft_consensus.cc:2379] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
W20260806 04:01:30.041538 4239 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260806 04:01:30.044482 4003 consensus_queue.cc:260] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.045148 3836 raft_consensus.cc:3032] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:30.045650 4000 consensus_queue.cc:1046] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 04:01:30.040647 32322 catalog_manager.cc:5495] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:30.018113 3811 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.047829 32322 catalog_manager.cc:5495] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:30.074620 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:30.052404 4098 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.115201 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.038501 32738 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
W20260806 04:01:30.051230 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 130161us
W20260806 04:01:30.117244 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 84368.1us
I20260806 04:01:30.117558 4000 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:30.118050 32553 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:30.118155 4000 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.119536 32553 leader_election.cc:398] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5.
I20260806 04:01:30.119839 32553 leader_election.cc:401] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 04:01:30.116427 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:30.120954 32553 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:30.121578 4008 raft_consensus.cc:2781] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:30.122011 4008 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:30.122056 3811 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.122560 4008 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:30.123572 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 74938.7us
I20260806 04:01:30.123054 3933 consensus_queue.cc:1046] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
W20260806 04:01:30.125248 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 100712us
I20260806 04:01:30.126015 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:30.122049 4007 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.126449 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:30.127583 3791 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:30.048499 4200 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.128542 32738 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:01:30.129891 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 149444us
I20260806 04:01:30.131047 1136 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.128070 3791 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:30.121513 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:30.131004 4012 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:30.132922 3791 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.133301 1188 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:30.134284 32553 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:30.024598 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:30.135164 3933 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:30.135389 32740 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:30.115286 32322 catalog_manager.cc:5495] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 3, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 3 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:30.136363 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:30.132004 3174 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:30.032634 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.116307 4098 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:30.077030 3698 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.126842 1137 leader_election.cc:398] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5.
I20260806 04:01:30.152472 1137 leader_election.cc:401] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 04:01:30.141058 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.053084 32749 raft_consensus.cc:2396] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:01:30.117230 32427 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.057039 32604 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
W20260806 04:01:30.149214 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 60021.2us
I20260806 04:01:30.136708 32322 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:30.069039 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:30.080037 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.045603 3836 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:30.155490 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:30.157843 32605 raft_consensus.cc:2379] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:30.157958 32469 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:30.157400 32610 raft_consensus.cc:2379] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:30.159710 32612 raft_consensus.cc:1226] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:30.033612 3854 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:30.064574 3647 consensus_queue.cc:260] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.039726 4145 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:30.156152 4189 consensus_meta.cc:287] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.536s user 0.005s sys 0.006s
I20260806 04:01:30.176452 32615 raft_consensus.cc:1226] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:30.050904 4003 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:30.071695 3751 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:30.120280 4124 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:30.123123 4164 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:30.173070 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 46443.1us
I20260806 04:01:30.189702 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:30.170138 32322 catalog_manager.cc:5495] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 reported cstate change: term changed from 2 to 3, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 3 leader_uuid: "d4a238f055a44f459c62c224daa4d541" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:30.163744 3836 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.139741 32738 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:30.192945 4042 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:30.193542 4042 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.164215 3854 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:30.196280 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 55289.3us
I20260806 04:01:30.147212 3933 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:30.198230 32428 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.141006 4000 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.184926 1192 raft_consensus.cc:1226] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:30.161880 32737 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:30.204478 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:30.205291 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 88226.6us
I20260806 04:01:30.211040 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:30.192955 3836 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:30.186510 4089 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:30.216711 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:30.216475 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:30.196061 4200 raft_consensus.cc:3032] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:30.161567 4095 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260806 04:01:30.198524 3854 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.216866 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:30.213855 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:30.221441 32467 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:30.204938 4156 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.208112 3933 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.194497 3811 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:30.068965 4185 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.221843 32741 raft_consensus.cc:2379] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:30.223878 3647 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:30.060905 4068 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:30.195919 3751 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.229738 32553 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.220463 4200 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:30.250027 4200 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.210765 4105 consensus_queue.cc:1046] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260806 04:01:30.252570 4105 raft_consensus.cc:3032] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:30.252930 4105 raft_consensus.cc:729] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:30.253731 4105 consensus_queue.cc:260] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.255200 4105 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:30.241326 4165 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:30.242283 4042 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:30.242774 3926 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:30.256475 4165 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.244221 3800 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:30.260727 3800 raft_consensus.cc:686] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:30.261721 3800 consensus_queue.cc:237] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.245242 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:30.247364 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:30.264492 32748 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:30.264559 32467 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:30.266196 32683 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.267688 4042 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:30.268124 4042 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:30.268443 4042 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.271085 4016 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.248689 3698 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:30.251636 4139 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:30.272812 4139 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.251323 4200 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:30.237620 4089 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:30.198762 4145 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.296624 32553 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:30.283238 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:30.298993 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.300643 32553 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.283631 32427 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:30.288183 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 51414.1us
I20260806 04:01:30.295920 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:30.279371 4042 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.308612 32466 raft_consensus.cc:2379] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:30.279548 4012 raft_consensus.cc:2726] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:30.284046 4185 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.309792 4042 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:30.311311 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:30.275416 4139 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.313715 3790 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:30.314159 3854 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:30.318615 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 78174.1us
I20260806 04:01:30.323246 4089 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:30.326109 4089 raft_consensus.cc:686] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:30.327064 4089 consensus_queue.cc:237] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.282323 4068 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:30.279891 3926 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:30.305359 4193 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:30.334571 4193 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:30.334990 4193 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:30.334900 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:30.340850 32471 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.342077 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.258584 4165 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.290457 4156 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5.
I20260806 04:01:30.232817 4018 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.293191 4145 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.294843 3751 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.273430 4016 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:30.339558 4092 consensus_meta.cc:287] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.654s user 0.005s sys 0.009s
I20260806 04:01:30.347450 32551 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.354240 1187 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
W20260806 04:01:30.358454 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 74952us
I20260806 04:01:30.363662 32553 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.348237 4006 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:30.355922 3261 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:30.373512 4006 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:30.369726 3926 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:30.376053 32427 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:30.374027 4006 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:30.377094 3790 raft_consensus.cc:2781] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:30.381469 32466 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:30.380297 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:30.383845 4192 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:30.384351 4192 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:30.384694 4192 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.392474 32609 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:30.373222 3854 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:30.379191 4189 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:30.379947 4006 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:30.401855 32551 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:30.412074 32608 raft_consensus.cc:2396] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:01:30.398278 3790 raft_consensus.cc:686] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:30.465543 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 97280.5us
I20260806 04:01:30.416206 4098 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:30.452200 32471 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:30.378153 3261 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.469828 32683 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.398159 3926 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260806 04:01:30.502210 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.505556 3854 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:30.505872 1137 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.506832 1137 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.507668 4156 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:30.508913 4182 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:30.509369 4182 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:30.509701 4182 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.511111 3790 consensus_queue.cc:237] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.485591 4165 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:30.449314 3933 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.518913 4156 raft_consensus.cc:686] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:30.492722 4018 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:30.529492 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 323206us
I20260806 04:01:30.530903 32682 leader_election.cc:398] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4.
I20260806 04:01:30.515829 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:30.549928 32427 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:30.553985 1136 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.534524 4165 raft_consensus.cc:686] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:30.568964 32682 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.568799 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:30.404214 4008 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.568933 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:30.575208 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:30.581413 1182 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:30.568742 3854 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:30.582082 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:30.571391 4145 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 04:01:30.594165 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 72056.8us
I20260806 04:01:30.587765 4156 consensus_queue.cc:237] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.598206 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.578034 3926 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:30.602244 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:30.602392 32553 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.586216 4095 raft_consensus.cc:2741] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 04:01:30.605764 32682 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:30.605393 1187 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:30.612488 32682 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.576681 3791 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.631537 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:30.632324 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 230553us
I20260806 04:01:30.632419 32427 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.613258 4098 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:30.640035 32551 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.613050 3751 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:30.592160 4042 consensus_queue.cc:1046] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260806 04:01:30.642544 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:30.702523 3751 raft_consensus.cc:3032] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:30.702751 32467 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.704104 3933 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.707350 4042 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:30.707947 4042 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.709429 3791 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.709898 4042 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.711020 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:30.711527 32602 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:30.642889 4250 log.cc:924] Time spent T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
W20260806 04:01:30.643658 4251 log.cc:924] Time spent T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260806 04:01:30.653589 4182 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.656411 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:30.656572 4008 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.717170 4182 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:30.720871 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:30.722919 4252 log.cc:924] Time spent T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
W20260806 04:01:30.660656 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 57473.6us
I20260806 04:01:30.662052 4189 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:30.665346 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.667546 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:30.668609 32471 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:30.670486 32609 raft_consensus.cc:1226] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 2. Current term is 3. Ops: []
W20260806 04:01:30.670855 4244 log.cc:924] Time spent T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
I20260806 04:01:30.673167 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.674556 4012 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:30.675563 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.738234 32615 raft_consensus.cc:2379] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:30.676898 4007 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.550s user 0.009s sys 0.001s
I20260806 04:01:30.675462 32473 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:30.678224 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.679983 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.680416 4165 consensus_queue.cc:237] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.682178 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:30.696866 4245 log.cc:924] Time spent T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20260806 04:01:30.697391 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 48506.7us
W20260806 04:01:30.732744 3174 consensus_meta.cc:287] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.595s user 0.003s sys 0.008s
I20260806 04:01:30.736779 32467 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:30.699299 4016 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:30.614506 3926 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.621961 4095 raft_consensus.cc:2781] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:30.618721 4145 raft_consensus.cc:686] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:30.734388 32603 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:30.739499 32743 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:30.742223 32605 raft_consensus.cc:2379] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:30.697690 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 81641.3us
I20260806 04:01:30.700503 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:30.746304 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 132718us
I20260806 04:01:30.747656 1137 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:30.724808 3751 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:30.767030 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:30.799157 32682 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:30.750379 4095 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:30.746191 4007 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5.
I20260806 04:01:30.746660 3174 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:30.797994 4008 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:30.810819 32682 leader_election.cc:302] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:30.808145 1136 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:30.805725 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.805902 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:30.806548 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.812363 32603 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:30.813355 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.814211 32603 raft_consensus.cc:2396] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:01:30.803231 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:30.816241 32614 raft_consensus.cc:2379] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:30.731261 4193 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.768967 4189 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:30.798269 4185 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:30.782615 4016 raft_consensus.cc:686] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:30.752713 4182 consensus_queue.cc:1046] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260806 04:01:30.825779 32615 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.826330 32427 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.778795 4012 consensus_queue.cc:1046] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260806 04:01:30.806412 4042 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:30.809451 4095 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:30.842808 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:30.847635 1136 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:30.848354 1183 raft_consensus.cc:2396] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:01:30.847880 4016 consensus_queue.cc:237] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:30.851061 4256 log.cc:924] Time spent T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.233s user 0.004s sys 0.000s
I20260806 04:01:30.852630 3800 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:30.817071 4124 consensus_queue.cc:1046] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260806 04:01:30.824852 4185 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:30.833504 4266 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:30.830188 3174 raft_consensus.cc:2741] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
W20260806 04:01:30.843082 4105 consensus_meta.cc:287] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.588s user 0.011s sys 0.000s
I20260806 04:01:30.859138 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:30.879218 32477 raft_consensus.cc:2379] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 2: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:30.859643 4266 raft_consensus.cc:686] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:30.859813 3836 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:30.857824 4185 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:30.862057 3694 raft_consensus.cc:3032] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:30.866286 4182 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:30.844437 4189 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:30.870472 4098 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:30.871572 4258 raft_consensus.cc:3032] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:30.871801 3261 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:30.822655 3751 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.846733 3791 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260806 04:01:30.833495 4145 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.873472 32615 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:30.850859 4193 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:30.874018 4192 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.846472 4284 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:30.895533 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:30.898442 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:30.857441 4068 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.529s user 0.004s sys 0.007s
W20260806 04:01:30.895084 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 74168.7us
I20260806 04:01:30.831830 4008 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.907821 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:30.910676 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:30.916980 32748 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:30.913431 32604 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:30.872973 3926 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.855707 4006 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.897995 3694 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:30.921155 1137 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:30.909549 4000 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:30.920998 32551 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:30.911217 4258 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:30.937422 32427 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:30.923385 3836 raft_consensus.cc:686] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:30.925406 4098 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:30.949493 3694 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.950196 4189 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:30.932765 4254 consensus_queue.cc:1046] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260806 04:01:30.951599 4254 raft_consensus.cc:3032] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:30.932823 4139 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:30.915244 4182 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.951905 4254 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:30.953095 4254 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.954368 4254 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:30.955053 4098 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.955499 4095 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.957425 4098 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.958302 4095 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.958439 4008 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:30.959580 4003 consensus_meta.cc:287] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.777s user 0.011s sys 0.000s
I20260806 04:01:30.960176 32739 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:30.960692 3698 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:30.961097 3698 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:30.961580 3698 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:30.963685 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:30.933064 4000 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:30.928993 3751 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:31.009393 3790 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:31.016770 3790 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.018867 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:30.996394 4182 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:31.020534 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:31.021165 32746 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.021908 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.000221 4018 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:31.023511 3790 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:30.964332 4266 consensus_queue.cc:237] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:30.976297 3926 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:30.927109 3174 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260806 04:01:31.030725 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.035802 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:31.036525 32471 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.052645 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:31.053258 4000 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:31.001888 3836 consensus_queue.cc:237] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.054757 4000 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.055264 4000 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.057354 4000 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:31.002470 32682 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:31.004282 32741 raft_consensus.cc:1226] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:31.006116 3800 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 04:01:31.059000 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 56618.3us
I20260806 04:01:31.007417 4006 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.002705 4258 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.003705 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:31.010442 3698 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.060931 3811 consensus_queue.cc:1046] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:31.059244 3800 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:30.961148 4192 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:31.007757 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 80077.2us
I20260806 04:01:31.063840 32427 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.065266 3790 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:31.053396 32745 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:31.065910 3790 consensus_queue.cc:1046] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:31.066998 3790 raft_consensus.cc:3032] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:31.067286 3790 raft_consensus.cc:729] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:31.068243 3790 consensus_queue.cc:260] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.065974 4192 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.069593 3790 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:31.069335 4192 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.071332 4192 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.072137 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.062647 3800 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.074127 3694 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:31.074371 3800 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:31.061573 3698 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:31.015731 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.001186 32748 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:31.065414 32427 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.080790 32427 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.081697 4124 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:31.082594 3697 consensus_queue.cc:1046] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:31.061937 3811 raft_consensus.cc:3032] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:31.061154 4258 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:31.085278 32608 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
W20260806 04:01:30.997632 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 79188.3us
I20260806 04:01:31.077206 32745 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:31.083351 4124 raft_consensus.cc:686] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:31.085314 32427 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.088065 32551 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.059828 32682 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.092833 32743 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:31.101815 32553 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:31.077100 4185 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.106806 32471 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:31.100216 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.114167 32469 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:31.115585 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.127372 32746 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
W20260806 04:01:31.137279 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 49310.5us
I20260806 04:01:31.138206 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:31.139158 32553 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:31.140897 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:31.141057 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 219984us
I20260806 04:01:31.083892 3811 raft_consensus.cc:729] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:31.143767 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 84056.9us
I20260806 04:01:31.143774 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.147682 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:31.150094 32615 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:31.150630 32553 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:31.159655 32467 raft_consensus.cc:2379] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 8: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:31.113767 4000 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:31.113745 4018 raft_consensus.cc:2741] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260806 04:01:31.174054 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.123152 4192 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:31.182739 32746 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:31.136488 4325 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:31.171968 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:31.186533 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 101296us
I20260806 04:01:31.144562 4006 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:31.190191 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.149981 4068 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:31.136442 3697 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:31.195606 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:31.187125 32612 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:31.196175 32745 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:31.180934 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.188665 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.186187 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:31.199220 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:31.193724 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.114190 3926 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260806 04:01:31.210045 4273 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.210803 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:31.210552 4273 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.211314 32746 raft_consensus.cc:1226] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:31.205384 4193 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:31.205914 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.207267 4016 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:31.216461 4016 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.207604 4007 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.218937 4007 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.220280 4018 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:31.220721 4018 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:31.221094 4018 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:31.221659 4007 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.208143 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.208395 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.225082 4016 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.225664 4016 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:31.226091 4016 raft_consensus.cc:686] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:31.226961 4016 consensus_queue.cc:237] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.228145 3836 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.228632 3836 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.229182 32745 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:31.207777 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.207397 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.233008 1187 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:31.175164 3791 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:31.186818 4200 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:01:31.236604 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 55759.6us
I20260806 04:01:31.184386 4192 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:31.237051 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 86506.2us
I20260806 04:01:31.243566 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.243782 32553 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:31.244369 32612 raft_consensus.cc:2379] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:31.243227 4165 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.247109 32608 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:31.247752 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.248381 32614 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:31.248682 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.249303 32604 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:31.249536 32614 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 04:01:31.249869 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.251278 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:31.252307 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.255688 4165 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:31.256397 3926 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:01:31.257807 3836 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:31.258817 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.259421 32473 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:31.195421 4145 raft_consensus.cc:3032] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:31.198576 3933 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:31.190541 4006 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:31.231420 32427 connection.cc:232] RPC call timeout handler was delayed by -1.2555316579528153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1967s for a response
I20260806 04:01:31.236559 4000 raft_consensus.cc:686] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:31.238852 4200 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.284780 4200 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.289892 4258 consensus_queue.cc:1046] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:31.290412 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.292133 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.293025 1184 raft_consensus.cc:2379] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:31.294294 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.276381 4006 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.296767 4189 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:31.297327 4189 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.299489 4189 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.300350 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.300762 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.301855 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.302963 32427 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.303867 32746 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:31.305797 4008 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:31.306686 3933 raft_consensus.cc:686] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:31.307618 3933 consensus_queue.cc:237] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.274519 32602 raft_consensus.cc:1226] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:31.265481 32615 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:31.271796 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.274852 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.244367 4192 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:31.279353 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.313898 1186 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:01:31.229839 3811 consensus_queue.cc:260] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.314877 3811 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:31.261633 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.317370 4185 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.318305 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:31.318575 4273 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.319072 32469 raft_consensus.cc:2379] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:31.310671 4092 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.319648 4000 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.236122 4325 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.311637 32603 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:31.321955 3926 raft_consensus.cc:686] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:31.238613 4193 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:31.322242 32612 raft_consensus.cc:1261] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:01:31.302095 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.320596 32609 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:31.323848 1185 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:31.319761 4092 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.328634 1186 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:31.298528 1137 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:31.245623 3791 raft_consensus.cc:686] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:31.298061 4258 raft_consensus.cc:3032] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:31.333516 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:31.338927 32604 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:31.266597 4145 raft_consensus.cc:729] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:31.341621 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.341850 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.344224 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.344998 32471 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:31.346107 32553 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:31.310550 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.343648 1186 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:31.356901 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.358664 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.360507 4322 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:31.362298 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.362989 1184 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:31.360728 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.340927 4091 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:31.364391 1182 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:31.364605 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.364777 4091 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.373090 32473 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:31.322412 4325 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.381461 1183 raft_consensus.cc:1226] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 04:01:31.381788 4042 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:31.333590 3751 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:31.384917 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.340960 3836 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:31.338246 4258 raft_consensus.cc:729] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:31.402676 32682 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:31.403124 32551 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:31.372332 3790 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:31.410768 4327 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:31.366815 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.381592 4254 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:31.268618 4124 consensus_queue.cc:237] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.415356 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.329936 4310 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.132s
I20260806 04:01:31.419046 32551 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:31.422376 1190 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:31.427399 32551 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:31.427412 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.385433 32602 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:31.423965 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.429119 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.430128 32551 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.359223 4193 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.436266 1192 raft_consensus.cc:2379] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:31.408677 4042 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:31.415982 4327 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.418263 3836 raft_consensus.cc:686] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:31.418135 4007 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.417956 4003 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:31.421109 4008 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:31.424937 4273 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.421592 3790 raft_consensus.cc:686] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:31.420548 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.453804 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.447847 4042 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.434028 4098 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:31.454291 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.405402 3751 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.473012 3751 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.473780 32745 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:31.473037 4091 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:31.461138 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.443094 4329 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:31.475688 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:31.454820 4007 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.448017 4258 consensus_queue.cc:260] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.477993 4258 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:31.478235 4007 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.464066 4264 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.480535 4264 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.481087 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.482550 4264 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:31.449499 3791 consensus_queue.cc:237] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.486467 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.465059 32743 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:01:31.452613 4003 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.468709 4092 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:31.461970 4273 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.470734 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.489564 1185 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:31.490072 4273 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.452020 3933 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:31.490617 1190 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:31.487990 4003 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.490835 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.494776 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.495265 4328 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.487483 32604 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:31.487948 32743 raft_consensus.cc:595] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:31.497215 32743 raft_consensus.cc:603] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:31.490571 1182 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:31.491916 4042 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.492676 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.503523 32467 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:31.495437 32748 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:31.505173 32553 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:31.449327 4327 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:31.499294 1192 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:31.498734 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:31.511505 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 81447.1us
I20260806 04:01:31.513774 32551 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:31.514182 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 54924.3us
I20260806 04:01:31.514905 1136 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.517474 1137 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:31.510567 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 97369.2us
I20260806 04:01:31.519640 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.521005 32551 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.526453 32682 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:31.531729 1190 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:31.532248 32609 raft_consensus.cc:1226] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 04:01:31.534111 1136 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.538925 1186 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:31.540401 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.540833 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:31.544552 32682 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:31.544929 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.545672 32604 raft_consensus.cc:2379] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:31.545580 32682 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:31.513376 4007 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:31.546701 32604 raft_consensus.cc:1261] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260806 04:01:31.548388 4313 log.cc:924] Time spent T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20260806 04:01:31.548709 3698 consensus_queue.cc:1046] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:31.549436 4091 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:31.580466 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.544538 4105 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:31.441428 3926 consensus_queue.cc:237] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.551766 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.587880 32603 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:31.551945 4095 raft_consensus.cc:2781] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 04:01:31.552626 4309 log.cc:924] Time spent T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260806 04:01:31.555279 1188 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:31.547320 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.556321 1186 raft_consensus.cc:2379] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 6: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:31.495838 4328 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.559605 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.557080 4007 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.559491 32551 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.579664 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:31.603658 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 44247.1us
I20260806 04:01:31.545701 4310 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20260806 04:01:31.582507 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 278546us
I20260806 04:01:31.581202 32604 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:31.588578 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.605826 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.606362 32612 raft_consensus.cc:2396] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 04:01:31.534762 4008 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:31.536723 4266 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:31.542869 4092 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:31.542076 4329 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:31.505267 4018 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.597822 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:31.608354 4325 raft_consensus.cc:2741] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 04:01:31.610018 4018 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:31.511073 4006 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.610671 4266 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.611084 4266 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:31.612094 4006 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.599730 1190 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:31.613209 32605 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:31.615104 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:31.615919 32743 raft_consensus.cc:2354] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6.
I20260806 04:01:31.617762 32467 raft_consensus.cc:1226] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:31.602682 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.620523 32553 leader_election.cc:398] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6.
I20260806 04:01:31.613241 3790 consensus_queue.cc:237] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.558486 4182 consensus_queue.cc:1046] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260806 04:01:31.603317 32610 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:31.605181 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.604439 32551 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:31.590924 3836 consensus_queue.cc:237] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.446702 4145 consensus_queue.cc:260] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.624238 4329 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:31.624914 4145 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:31.624965 4329 consensus_queue.cc:1046] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:31.626317 4329 raft_consensus.cc:3032] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:31.626668 4329 raft_consensus.cc:729] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:31.627689 4329 consensus_queue.cc:260] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.628398 32602 raft_consensus.cc:1226] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:31.629251 4329 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.620872 32553 leader_election.cc:401] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:01:31.631978 4339 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:31.580503 4185 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.607834 32427 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:31.587447 1184 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:31.580408 4091 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.646644 1136 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:31.649509 32609 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
W20260806 04:01:31.656098 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 48312.8us
I20260806 04:01:31.664287 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:31.632815 4007 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:31.672423 1189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4f865676f884755987b8bd820489186"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:31.674654 1189 raft_consensus.cc:595] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.663225 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:31.672377 4285 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:31.652927 4042 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:31.644865 4273 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:31.648203 4095 raft_consensus.cc:686] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:31.635694 4018 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.681146 32682 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:31.636708 4008 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.652102 4189 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.665052 4091 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.655210 4006 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.655423 4322 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:31.664420 3697 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.677093 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.696597 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.665671 3800 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:31.693125 32745 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260806 04:01:31.694335 4285 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.681703 1183 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:31.700265 32745 pending_rounds.cc:76] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Aborting all ops after (but not including) 4
I20260806 04:01:31.720885 32745 pending_rounds.cc:97] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260806 04:01:31.721035 32471 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:31.699182 32473 raft_consensus.cc:1226] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:31.698817 4008 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.651232 3647 consensus_queue.cc:1035] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260806 04:01:31.693737 4254 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:31.707135 4323 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.708819 4307 raft_consensus.cc:3032] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:31.710196 4273 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.761108 4307 raft_consensus.cc:729] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:31.761497 4273 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:31.762413 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 130678us
I20260806 04:01:31.709861 4200 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.768613 4200 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.769150 4200 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.769600 4200 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.771548 4200 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.710911 4092 raft_consensus.cc:3032] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:31.772104 4092 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:31.773011 4092 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.774436 4092 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:31.712785 1187 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 04:01:31.777218 1187 pending_rounds.cc:76] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Aborting all ops after (but not including) 5
I20260806 04:01:31.777513 1187 pending_rounds.cc:97] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 04:01:31.777737 1187 raft_consensus.cc:2866] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:31.711879 4285 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:31.672677 4139 consensus_queue.cc:1046] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260806 04:01:31.721172 32745 raft_consensus.cc:2866] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:31.698451 4332 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:31.725818 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.786554 32474 raft_consensus.cc:2379] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:31.650277 4192 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:01:31.698190 4314 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.455s user 0.000s sys 0.003s
I20260806 04:01:31.806357 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:31.692663 4042 raft_consensus.cc:686] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:31.729992 4185 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:31.694712 4000 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:31.762022 4307 consensus_queue.cc:260] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.699776 4018 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.700203 3800 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:31.807719 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:31.807740 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 74812.6us
I20260806 04:01:31.723156 4189 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.698654 3697 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:31.809929 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.815110 32683 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.682586 4193 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:31.823089 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 126528us
I20260806 04:01:31.824235 32749 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:31.731626 4254 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.731293 4006 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:31.826062 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 179486us
I20260806 04:01:31.828225 32427 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.828919 32553 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:31.819120 3647 raft_consensus.cc:3032] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:31.848167 3647 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:31.849121 4192 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:31.849062 3647 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.813206 4000 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:31.850265 3647 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:31.850484 4018 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:31.832814 4042 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.826675 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:01:31.857758 32427 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.831668 1136 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:31.832772 4328 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.825232 4185 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.835510 4006 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.825307 3800 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:31.841753 4326 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:31.842489 4284 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:31.846076 4189 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:31.846740 4105 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:31.896273 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:31.897001 32473 raft_consensus.cc:2379] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 8: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:31.858263 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:31.897073 32615 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:31.850386 4000 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:31.850281 3697 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:31.893277 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.855463 4254 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:31.898556 32471 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:31.897701 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:31.880378 4098 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:01:31.886518 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 54923.7us
I20260806 04:01:31.927254 32467 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:31.926290 32469 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:31.888680 4165 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:31.935544 32551 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:31.929852 32553 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:31.903036 32427 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.934823 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:31.938417 32609 raft_consensus.cc:1261] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:01:31.899112 4285 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:31.893424 4018 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:31.894412 4284 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:31.885612 3694 consensus_queue.cc:1046] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260806 04:01:31.817348 4095 consensus_queue.cc:237] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.948761 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.927953 4098 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.901605 4326 raft_consensus.cc:686] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:31.955591 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:31.962606 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 69768.2us
I20260806 04:01:31.972787 32427 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.888460 4185 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.933146 4339 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6.
I20260806 04:01:31.949308 4189 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:31.899451 4016 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260806 04:01:31.890130 4266 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:31.968999 32615 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:31.947829 3791 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:31.924487 3751 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260806 04:01:31.988113 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.989562 1187 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:32.004606 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.990586 4185 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:31.986687 4189 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:31.981784 4018 raft_consensus.cc:686] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:31.996853 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:31.952234 4310 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:32.022805 32427 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:31.954959 4284 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:31.997452 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:31.952962 3926 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260806 04:01:31.977957 3790 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:31.977813 4098 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:32.005306 32604 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:32.006026 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:32.041031 32467 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:32.006794 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:31.978546 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:31.969877 4193 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:32.012550 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.015751 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.054612 1185 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 8: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:32.001463 32553 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:32.026080 32427 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:31.946762 4165 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.027940 32471 raft_consensus.cc:1261] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:31.985493 3697 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:32.020490 4189 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:32.037127 32605 raft_consensus.cc:1261] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 04:01:32.070541 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 44562.5us
I20260806 04:01:31.988397 4003 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:32.039929 32741 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:32.041034 1186 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 04:01:32.043229 32603 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:32.042523 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:32.004683 3791 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:32.076328 32608 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:32.050199 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.004421 4332 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:32.053485 32749 raft_consensus.cc:1261] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:32.054466 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:32.026616 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.035964 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:32.060525 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 59153.6us
I20260806 04:01:32.055065 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.062871 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:32.064149 3261 consensus_queue.cc:1035] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:32.091663 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:32.050938 4124 raft_consensus.cc:2741] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 04:01:32.068272 4329 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:32.068885 4193 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:31.985493 4008 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.071221 32427 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:31.986852 3836 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260806 04:01:32.078306 32682 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:32.073547 3790 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:32.084493 1190 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:32.086329 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:32.077315 1189 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:32.041131 4310 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.094512 32469 raft_consensus.cc:1261] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:32.065686 4165 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.102772 32746 raft_consensus.cc:1226] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
W20260806 04:01:32.088418 4352 log.cc:924] Time spent T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.261s user 0.001s sys 0.001s
I20260806 04:01:32.106765 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:32.086248 4327 consensus_meta.cc:287] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.565s user 0.009s sys 0.000s
I20260806 04:01:32.084124 1192 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:32.110018 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:32.080581 3791 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:32.080253 3697 raft_consensus.cc:686] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:32.115708 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:32.126927 32739 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:32.128515 1186 raft_consensus.cc:2379] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:32.097450 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.129751 32682 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:32.050779 4344 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260806 04:01:32.120823 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:32.105430 4329 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:32.161296 32745 raft_consensus.cc:1261] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:01:32.164885 32427 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:32.139279 1187 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:32.060922 4007 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.129794 32428 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:32.247963 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 119649us
I20260806 04:01:32.158603 1188 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:32.116251 4322 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:32.177955 4364 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:32.267707 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:32.268421 32615 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:32.180989 4003 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:32.181459 4370 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260806 04:01:32.182967 4379 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:32.183462 4346 consensus_queue.cc:1035] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 04:01:32.184703 4361 log.cc:924] Time spent T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20260806 04:01:32.167301 4325 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 04:01:32.185283 4353 log.cc:924] Time spent T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.337s user 0.001s sys 0.000s
I20260806 04:01:32.167775 4329 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:32.199728 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:32.200357 4145 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:32.200691 4328 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:32.202577 4006 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:32.205731 4266 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:32.214233 4018 consensus_queue.cc:237] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.119712 4339 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 04:01:32.130770 4355 log.cc:924] Time spent T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.249s user 0.001s sys 0.000s
I20260806 04:01:32.118660 4139 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:32.225147 4091 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:32.255831 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 82655.4us
I20260806 04:01:32.343892 4145 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:32.262058 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:32.255437 1192 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:32.071856 4326 consensus_queue.cc:237] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.095469 4332 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.129988 4068 consensus_queue.cc:1035] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.130s
I20260806 04:01:32.415980 4145 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:32.325191 4092 consensus_meta.cc:287] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.550s user 0.008s sys 0.004s
I20260806 04:01:32.469204 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:32.469857 32746 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:32.470515 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:32.330161 4310 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:32.335708 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 72020us
I20260806 04:01:32.327407 4000 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.266371 3790 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:32.131750 4327 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:32.372252 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 354670us
I20260806 04:01:32.376576 4003 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:32.260056 4322 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:32.403118 32615 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:32.335294 4325 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:32.501384 32748 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
W20260806 04:01:32.477813 4323 consensus_meta.cc:287] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.695s user 0.006s sys 0.004s
W20260806 04:01:32.505070 3647 consensus_meta.cc:287] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.654s user 0.010s sys 0.000s
I20260806 04:01:32.505606 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:32.506287 32470 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:01:32.510078 4364 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:32.510762 32738 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:32.513952 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.518203 4091 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:32.518484 4098 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.521986 4332 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.524955 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:32.525801 4003 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:32.526854 3791 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.528167 4325 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:32.448225 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:32.531816 32474 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
W20260806 04:01:32.532224 4273 consensus_meta.cc:287] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.770s user 0.010s sys 0.000s
I20260806 04:01:32.340962 4379 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.532624 4273 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.533990 4379 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.440490 3811 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:32.535209 4273 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:32.438444 4380 log.cc:924] Time spent T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20260806 04:01:32.410290 4139 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:32.536496 4007 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.536799 4139 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:32.537117 4323 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:32.537856 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:32.538483 32740 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:32.320405 3697 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.480367 3790 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:32.540719 3790 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:32.444574 4284 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:32.497530 4322 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:32.498206 4193 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:32.549191 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 44474.2us
I20260806 04:01:32.549595 32609 raft_consensus.cc:2354] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9.
I20260806 04:01:32.471094 32471 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:32.402350 4200 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260806 04:01:32.472631 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:32.475519 1192 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 04:01:32.476641 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 207112us
I20260806 04:01:32.553697 32739 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:32.421128 4328 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:32.408325 4006 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.489063 3751 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:32.495489 4389 log.cc:924] Time spent T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
W20260806 04:01:32.531733 4254 consensus_meta.cc:287] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.596s user 0.006s sys 0.000s
I20260806 04:01:32.499959 32682 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:32.500505 1136 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:32.504981 32553 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
W20260806 04:01:32.600625 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 95717.7us
I20260806 04:01:32.602373 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:32.427934 4360 consensus_queue.cc:1035] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260806 04:01:32.576797 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:32.490536 4092 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:32.578028 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:32.610011 32469 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:32.497224 32745 raft_consensus.cc:1261] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260806 04:01:32.578881 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 379229us
I20260806 04:01:32.519865 4327 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.527854 4003 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:32.621515 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:32.606504 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:32.572980 3933 consensus_meta.cc:287] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.072s user 0.011s sys 0.000s
I20260806 04:01:32.579807 4323 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:32.612362 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:32.614976 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:32.571975 3800 consensus_meta.cc:287] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.676s user 0.009s sys 0.001s
I20260806 04:01:32.619350 32749 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:32.617394 32746 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:32.495903 4369 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.121s
W20260806 04:01:32.625458 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 125584us
I20260806 04:01:32.626504 32471 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
W20260806 04:01:32.608273 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 107855us
I20260806 04:01:32.646005 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:32.646665 32609 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:32.647069 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:32.623006 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:32.647857 32748 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:32.634258 32683 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:32.648334 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:32.648890 32471 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:32.610256 4007 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:32.609187 4193 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:32.650288 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:32.653590 1137 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:32.647156 32551 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:32.656486 32478 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 9
I20260806 04:01:32.655817 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:32.657351 32604 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:32.618217 3694 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:32.658923 32551 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:32.619091 4310 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:32.665606 32748 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:32.665977 32615 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:32.674563 32471 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:32.675776 32553 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:32.677356 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:32.675690 32615 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:32.599603 4328 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:32.683168 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.633217 3647 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:32.599828 3751 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.686913 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:32.633219 3933 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:32.656235 4284 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:32.686229 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:32.604987 4254 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.656195 4193 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:32.684103 1192 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:32.657799 4098 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:32.707540 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 73357.4us
I20260806 04:01:32.695837 4339 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:32.678653 4264 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:32.723037 32605 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:32.699985 4370 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:32.668624 3694 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:32.729493 4193 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:32.619630 4092 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.730389 1185 raft_consensus.cc:1226] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:32.655678 4003 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.053s
I20260806 04:01:32.688288 4007 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 8 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:32.659665 4008 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:32.738008 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:32.740314 4006 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.650599 3800 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.745210 32682 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
W20260806 04:01:32.791613 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 61411.9us
W20260806 04:01:32.792531 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 47413.1us
I20260806 04:01:32.793567 32682 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:32.745607 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:32.790580 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:32.794938 1186 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:32.797413 32471 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260806 04:01:32.797778 32471 pending_rounds.cc:76] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 3
I20260806 04:01:32.797989 32471 pending_rounds.cc:97] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20260806 04:01:32.798215 32471 raft_consensus.cc:2866] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 04:01:32.746204 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 134717us
I20260806 04:01:32.669085 4258 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.060s
I20260806 04:01:32.669939 4327 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260806 04:01:32.627905 4399 consensus_queue.cc:1046] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
I20260806 04:01:32.778380 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:32.804960 32427 leader_election.cc:398] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9.
I20260806 04:01:32.805307 32427 leader_election.cc:401] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:32.778705 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:32.684443 4310 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.731376 3791 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.650784 4323 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
W20260806 04:01:32.737574 4366 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.658s user 0.003s sys 0.000s
I20260806 04:01:32.728291 4373 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:32.747504 3694 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:32.805997 1187 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 7
I20260806 04:01:32.805135 1188 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:32.780714 4264 raft_consensus.cc:686] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:32.793030 4068 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:32.784140 4339 raft_consensus.cc:686] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:32.821467 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:32.835399 32615 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:32.836851 32682 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:32.792845 4284 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:32.805575 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:32.793363 4370 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 8 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:32.839844 32553 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:32.804603 4098 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:32.802999 4000 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.803741 1184 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:32.846230 32604 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:01:32.837630 4068 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:32.791183 4095 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260806 04:01:32.902686 32427 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:32.900807 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:32.845077 4273 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:32.906591 1187 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:32.922578 32471 raft_consensus.cc:2379] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:32.855890 1186 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:32.928721 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:32.930133 1188 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
W20260806 04:01:32.931890 1190 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.010s sys 0.004s
W20260806 04:01:32.933075 1190 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s user 0.010s sys 0.004s
I20260806 04:01:32.859748 4008 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:32.931407 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:32.852403 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.830032 4200 consensus_queue.cc:1035] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
I20260806 04:01:32.911972 4068 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:32.911046 3647 raft_consensus.cc:686] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:32.841558 4018 consensus_queue.cc:1035] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260806 04:01:32.933151 4000 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:32.852749 4284 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:32.968132 4328 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:32.893348 4098 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:32.937091 4258 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:32.981668 32551 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:32.966887 4008 raft_consensus.cc:686] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:32.974287 4000 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:32.978442 4165 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:32.978281 4254 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:33.001327 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:32.996436 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:32.999985 32551 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:33.003727 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:33.034376 1185 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:33.038728 1189 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:32.995422 4419 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
W20260806 04:01:32.994733 4145 consensus_meta.cc:287] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.516s user 0.009s sys 0.000s
I20260806 04:01:32.923483 4007 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:32.991508 4092 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:32.985126 3751 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:32.994050 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 610) took 1341 ms. Trace:
I20260806 04:01:32.995446 4000 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:32.996227 4417 log.cc:924] Time spent T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20260806 04:01:32.996274 4424 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.003670 4328 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:33.089790 32474 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:33.042779 4165 raft_consensus.cc:686] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:32.830586 4373 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.098321 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:01:33.094484 4409 log.cc:924] Time spent T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.320s user 0.001s sys 0.001s
I20260806 04:01:32.992262 4258 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.045552 4341 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:32.970731 4339 consensus_queue.cc:237] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.000342 3697 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.090s
W20260806 04:01:33.023715 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 220929us
I20260806 04:01:33.109637 32614 raft_consensus.cc:2396] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260806 04:01:33.109894 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 109984us
I20260806 04:01:33.043762 4370 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.091234 3800 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.167940 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:33.168599 32746 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:32.985487 4264 consensus_queue.cc:237] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.132618 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:33.097604 4310 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.154276 4285 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:33.128405 4423 log.cc:924] Time spent T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.297s user 0.000s sys 0.001s
I20260806 04:01:33.048856 4402 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.273s
I20260806 04:01:33.125753 4006 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:33.152495 32478 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
I20260806 04:01:33.160887 32551 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:33.187790 32614 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:33.113780 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:33.188674 1185 raft_consensus.cc:2396] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:01:33.164155 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:33.055294 3647 consensus_queue.cc:237] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:33.133499 1189 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:33.139267 4428 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:33.157955 3261 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.141353 4323 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.157156 4092 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:33.156124 4328 raft_consensus.cc:686] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:33.129263 4145 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:33.209720 4307 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:33.210947 4145 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:33.211066 4440 log.cc:924] Time spent T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260806 04:01:33.211622 32477 raft_consensus.cc:2354] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4.
I20260806 04:01:33.189441 32612 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:33.192183 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:33.193895 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.208734 32614 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260806 04:01:33.184566 4189 consensus_meta.cc:287] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.086s user 0.013s sys 0.001s
I20260806 04:01:33.176823 4341 raft_consensus.cc:686] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:33.183749 4284 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.128881 4008 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.178346 3751 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
W20260806 04:01:33.204913 4403 log.cc:924] Time spent T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.317s user 0.000s sys 0.001s
I20260806 04:01:33.206003 4453 consensus_queue.cc:1046] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
W20260806 04:01:33.207098 4450 log.cc:924] Time spent T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20260806 04:01:33.207849 4441 log.cc:924] Time spent T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20260806 04:01:33.208192 4436 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:33.193109 4098 leader_election.cc:288] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.208803 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 497) took 1214 ms. Trace:
I20260806 04:01:33.128732 32602 rpcz_store.cc:276] 0806 04:01:31.647967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:31.651718 (+ 3751us) service_pool.cc:227] Handling call
0806 04:01:31.684719 (+ 33001us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:31.723321 (+ 38602us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:31.728817 (+ 5496us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:31.812622 (+ 83805us) log.cc:842] Serialized 26 byte log entry
0806 04:01:31.928312 (+115690us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:31.938054 (+ 9742us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:31.944848 (+ 6794us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:32.682635 (+737787us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:32.973600 (+290965us) raft_consensus.cc:1671] finished
0806 04:01:32.975023 (+ 1423us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:32.989067 (+ 14044us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11834,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"spinlock_wait_cycles":15476608,"thread_start_us":67254,"threads_started":1,"wal-append.queue_time_us":151834}
I20260806 04:01:33.209133 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1749) took 1288 ms. Trace:
I20260806 04:01:33.255148 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:33.208678 4442 log.cc:924] Time spent T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260806 04:01:33.194712 4327 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:33.194199 3791 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:33.204879 4452 log.cc:924] Time spent T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20260806 04:01:33.143517 4424 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.225793 4341 consensus_queue.cc:237] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:33.227108 1185 raft_consensus.cc:1261] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:01:33.197249 4006 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:33.231880 32746 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:01:33.268412 32746 raft_consensus.cc:595] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:33.268762 32746 raft_consensus.cc:603] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:33.212122 4418 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9.
I20260806 04:01:33.186333 4285 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:33.254154 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 276358us
I20260806 04:01:33.214421 1189 raft_consensus.cc:2379] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:33.257993 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.254978 32743 rpcz_store.cc:276] 0806 04:01:31.994291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:31.994465 (+ 174us) service_pool.cc:227] Handling call
0806 04:01:31.994778 (+ 313us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:32.098025 (+103247us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:32.098055 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:32.433773 (+335718us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b2c00108868. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:32.434138 (+ 365us) log.cc:842] Serialized 26 byte log entry
0806 04:01:32.599145 (+165007us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:32.599188 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:32.599261 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:33.196814 (+597553us) raft_consensus.cc:1671] finished
0806 04:01:33.196839 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:33.208715 (+ 11876us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":44360,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":9491,"spinlock_wait_cycles":15205120,"thread_start_us":267440,"threads_started":2,"wal-append.queue_time_us":203652}
I20260806 04:01:33.266839 4006 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 7.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.202608 3261 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.215255 4092 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.222078 4189 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.283031 32683 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:33.261791 4429 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:33.244817 4284 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260806 04:01:33.292877 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:33.277532 4327 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.291016 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:33.281565 4139 consensus_meta.cc:287] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.742s user 0.010s sys 0.001s
I20260806 04:01:33.301162 32749 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:33.288004 4328 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.306819 4373 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.308286 4139 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.310437 4068 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:33.312384 32741 raft_consensus.cc:2379] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:33.312547 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:33.259477 4436 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.313450 32602 raft_consensus.cc:2354] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:33.313994 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2267) took 1199 ms. Trace:
I20260806 04:01:33.313927 4189 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.314611 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:33.315438 32602 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:33.316032 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 865) took 1298 ms. Trace:
I20260806 04:01:33.316478 1190 rpcz_store.cc:276] 0806 04:01:32.017295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.118908 (+101613us) service_pool.cc:227] Handling call
0806 04:01:32.119221 (+ 313us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:32.130328 (+ 11107us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:32.130360 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:32.232237 (+101877us) log.cc:842] Serialized 26 byte log entry
0806 04:01:32.934502 (+702265us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:32.934545 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:32.934610 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:33.315767 (+381157us) raft_consensus.cc:1671] finished
0806 04:01:33.315788 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:33.315927 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25088,"raft.queue_time_us":1805,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":10644,"thread_start_us":591616,"threads_started":1,"wal-append.queue_time_us":710815}
I20260806 04:01:33.318132 4424 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:33.318871 4325 consensus_meta.cc:287] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.790s user 0.011s sys 0.001s
I20260806 04:01:33.318697 4000 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.269649 3791 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:33.303864 3790 consensus_meta.cc:287] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.763s user 0.007s sys 0.006s
I20260806 04:01:33.320852 3790 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:33.322206 3261 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.323096 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:33.323704 32466 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:33.287420 4466 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.314477 32741 rpcz_store.cc:276] 0806 04:01:32.114875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.116209 (+ 1334us) service_pool.cc:227] Handling call
0806 04:01:33.313800 (+1197591us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54000a87d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:33.313934 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2656538368}
I20260806 04:01:33.302136 32612 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:33.320248 4092 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.325121 32467 raft_consensus.cc:1226] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:33.219573 4165 consensus_queue.cc:237] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.337046 4430 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:33.337940 4451 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:33.338543 4451 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.301229 4258 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.259262 32738 rpcz_store.cc:276] 0806 04:01:31.920502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:31.924757 (+ 4255us) service_pool.cc:227] Handling call
0806 04:01:33.208941 (+1284184us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b5400102a54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:33.209053 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1031672064}
I20260806 04:01:33.300844 4429 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.325680 4327 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.298476 4310 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.344936 4285 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:33.349340 3790 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.337257 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:33.301832 32740 consensus_meta.cc:287] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.763s user 0.011s sys 0.002s
I20260806 04:01:33.350637 32478 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:33.319407 4325 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.334465 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:33.330657 4457 log.cc:924] Time spent T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
W20260806 04:01:33.332960 4431 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.214s user 0.000s sys 0.001s
I20260806 04:01:33.344445 4310 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:33.280555 4322 consensus_meta.cc:287] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.734s user 0.011s sys 0.002s
I20260806 04:01:33.351611 32553 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:33.332495 32427 leader_election.cc:398] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:33.378371 32427 leader_election.cc:401] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 04:01:33.297132 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:33.368522 1185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:33.379696 1185 raft_consensus.cc:595] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:33.325488 4329 consensus_meta.cc:287] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.934s user 0.015s sys 0.000s
I20260806 04:01:33.370572 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:33.322206 3791 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:33.378966 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 46580.3us
W20260806 04:01:33.377091 4410 log.cc:924] Time spent T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
I20260806 04:01:33.403225 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:33.404377 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 04:01:33.368480 4430 raft_consensus.cc:686] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:33.401305 32604 raft_consensus.cc:1226] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:33.369127 32740 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:33.406240 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:33.412591 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:33.401711 4008 raft_consensus.cc:2781] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:33.350517 3694 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.427037 32466 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:33.424669 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:33.330600 4456 consensus_queue.cc:1035] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.086s
I20260806 04:01:33.433669 32553 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:33.403749 4000 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.403087 4139 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.433707 1185 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:33.435101 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:33.442049 1189 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:01:33.434653 4008 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:33.442916 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:33.350960 4466 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.446780 4466 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.390920 4322 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.450498 4322 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.451372 4400 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:33.452961 3694 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:33.437096 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:33.454497 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:33.455134 1187 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:33.455958 32604 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:33.458221 32475 raft_consensus.cc:1226] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:33.460227 4430 consensus_queue.cc:237] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:33.432337 3790 consensus_queue.cc:1046] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260806 04:01:33.462917 3790 raft_consensus.cc:3032] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:33.463375 3790 raft_consensus.cc:729] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:33.464026 4325 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.464447 4370 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:33.464697 4145 raft_consensus.cc:3032] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:33.465193 4145 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:33.452211 4400 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.461108 4451 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.439847 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.438134 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:33.406213 4200 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.417369 3791 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.466995 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:33.394905 4329 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:33.508033 32612 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:33.435576 4460 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:33.468395 4008 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:33.497875 1187 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:33.438911 4068 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:33.496273 1189 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:33.502321 4200 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.483495 32602 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:33.507270 1185 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:33.509589 32608 raft_consensus.cc:2354] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5.
I20260806 04:01:33.520954 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:33.521637 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:33.522238 32682 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:33.470654 4007 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:33.523149 32427 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:33.526288 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:33.486439 4016 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:01:33.510098 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 209995us
I20260806 04:01:33.465303 4370 raft_consensus.cc:686] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:33.516772 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:33.517750 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:33.528573 32428 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:33.532363 1189 raft_consensus.cc:2354] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:33.532517 32604 raft_consensus.cc:2396] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:01:33.529486 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:33.529877 32615 raft_consensus.cc:1226] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:33.533813 32466 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:33.534941 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:33.533792 32683 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:33.522787 1136 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:33.540411 32466 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:33.543483 4429 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.512502 4327 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.545380 32553 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:33.550982 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:33.551595 32683 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:33.553869 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:33.558226 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:33.559332 1185 raft_consensus.cc:2396] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:01:33.536414 3647 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:33.534718 4466 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:33.516109 3694 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 04:01:33.534951 4007 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.512359 3790 consensus_queue.cc:260] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:33.529610 4145 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.565678 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:33.544752 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:33.567188 4098 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.566901 4329 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:33.573271 4488 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:33.568825 4492 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260806 04:01:33.542157 4399 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:33.515405 3791 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.575114 1189 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:33.659140 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:33.557724 4451 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:33.667812 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:33.560513 4429 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:33.677780 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:33.611466 4193 consensus_meta.cc:287] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.821s user 0.011s sys 0.001s
I20260806 04:01:33.655807 3261 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:33.680395 32749 raft_consensus.cc:1226] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 04:01:33.574862 4327 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:33.680755 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:33.683375 1136 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:33.700440 32478 raft_consensus.cc:2354] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5.
I20260806 04:01:33.598519 4486 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:33.682210 1187 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:33.611877 4466 raft_consensus.cc:686] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:33.708698 4400 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.709894 4258 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:33.709314 4466 consensus_queue.cc:237] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:33.710489 4258 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:33.710996 4258 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:33.712632 4411 log.cc:924] Time spent T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.845s user 0.004s sys 0.000s
I20260806 04:01:33.712795 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:33.713651 1187 raft_consensus.cc:2379] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:33.714268 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1660) took 1268 ms. Trace:
I20260806 04:01:33.714938 1187 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:33.634557 4399 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:33.715673 4399 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:33.686751 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.598027 4436 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.680445 4432 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:33.796754 4432 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:33.681154 1184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 825) took 1669 ms (client timeout 2059 ms). Trace:
I20260806 04:01:33.691145 4007 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:33.798416 1184 rpcz_store.cc:269] 0806 04:01:32.011292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.021162 (+ 9870us) service_pool.cc:227] Handling call
0806 04:01:32.513834 (+492672us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4bce 000000000034aea9 00007fd172120a30 00007fd1721e67b4 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:33.680899 (+1167065us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54006dabd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:33.681031 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":665629824}
I20260806 04:01:33.581278 3647 raft_consensus.cc:686] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:33.801170 3647 consensus_queue.cc:237] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.604921 4310 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:33.694876 32471 raft_consensus.cc:1226] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:33.692268 32740 raft_consensus.cc:2379] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:33.652792 4329 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:33.695977 4416 log.cc:924] Time spent T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.724s user 0.002s sys 0.000s
I20260806 04:01:33.652331 4098 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.678601 4325 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:33.604846 4421 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:33.704098 32604 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:33.606830 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:33.741073 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 70606.8us
I20260806 04:01:33.747861 1187 raft_consensus.cc:1226] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:33.750310 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:33.714617 32473 rpcz_store.cc:276] 0806 04:01:32.446042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.447200 (+ 1158us) service_pool.cc:227] Handling call
0806 04:01:32.476481 (+ 29281us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:32.476800 (+ 319us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:32.476829 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:32.496225 (+ 19396us) log.cc:842] Serialized 26 byte log entry
0806 04:01:32.618996 (+122771us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:32.619034 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:32.619098 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:33.628551 (+1009453us) spinlock_profiling.cc:243] Waited 9.78 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:33.713985 (+ 85434us) raft_consensus.cc:1671] finished
0806 04:01:33.714011 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:33.714204 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":40376,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":100,"spinlock_wait_cycles":22488832,"thread_start_us":122440,"threads_started":1,"wal-append.queue_time_us":202546}
I20260806 04:01:33.652364 4488 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.755823 3791 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.757543 4200 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:33.745139 4486 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.761325 1190 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:33.717283 4429 raft_consensus.cc:686] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:33.769187 4451 raft_consensus.cc:2726] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:33.768455 4258 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.771095 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:33.702960 4193 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:33.706740 3261 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.716571 4370 consensus_queue.cc:237] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.832309 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:33.868034 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 105985us
I20260806 04:01:33.849926 1137 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:33.863969 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:33.865794 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.869354 32743 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:33.871750 1137 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:33.882268 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.890014 32610 raft_consensus.cc:2354] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4.
I20260806 04:01:33.891525 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.893249 32471 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:33.894127 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.865820 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:33.827692 4329 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.917768 32428 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:34.085114 4381 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:34.083211 4000 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:33.920097 4428 consensus_queue.cc:1046] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.167s
I20260806 04:01:34.088374 4428 consensus_queue.cc:1225] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer dd7344d85a154539b7f8c484f2a76772 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 04:01:34.088958 4381 raft_consensus.cc:3032] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:33.917913 32683 leader_election.cc:398] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5.
I20260806 04:01:34.089295 4381 raft_consensus.cc:729] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:34.087095 4000 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:34.090379 4381 consensus_queue.cc:260] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.090924 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1758) took 1631 ms. Trace:
I20260806 04:01:34.091672 4381 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:33.896849 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:34.097711 32478 raft_consensus.cc:2379] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:34.093950 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:34.096589 32683 leader_election.cc:401] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260806 04:01:34.085270 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 167610us
I20260806 04:01:33.888162 4386 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:01:34.111886 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 194091us
I20260806 04:01:34.112753 32683 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:33.882150 4400 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:34.116626 4185 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:33.894673 4424 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:34.124364 4185 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.091632 4000 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:34.093272 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:34.128329 4400 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.131512 4185 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:34.141556 4379 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:34.142434 4488 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:34.145589 4008 consensus_meta.cc:287] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.616s user 0.011s sys 0.000s
I20260806 04:01:34.127481 4329 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:34.147517 4258 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:34.149137 4496 log.cc:924] Time spent T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260806 04:01:34.149084 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:34.150372 3261 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:34.150357 32738 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:34.151018 4193 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:34.122490 4487 log.cc:924] Time spent T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.255s user 0.000s sys 0.000s
I20260806 04:01:34.152441 4400 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:34.158278 32682 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.237.130:37857 timed out after 1.515s (SENT) [suppressed 10 similar messages]
I20260806 04:01:34.140789 4486 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:34.178238 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:33.921818 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:33.890671 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 207376us
I20260806 04:01:34.199213 32471 raft_consensus.cc:2379] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:33.868016 4327 raft_consensus.cc:686] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:33.883512 4509 consensus_queue.cc:1046] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.322s
I20260806 04:01:34.205626 4509 consensus_queue.cc:1225] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer f7c202b562ce482db283463d7f02e136 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:01:34.207674 4509 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.208431 4509 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:33.949473 32743 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:34.091359 32737 rpcz_store.cc:276] 0806 04:01:32.459206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.464520 (+ 5314us) service_pool.cc:227] Handling call
0806 04:01:32.494356 (+ 29836us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:32.512009 (+ 17653us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:32.512040 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:32.512561 (+ 521us) log.cc:842] Serialized 26 byte log entry
0806 04:01:32.697245 (+184684us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:32.704264 (+ 7019us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:32.721517 (+ 17253us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:33.691540 (+970023us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:33.927721 (+236181us) raft_consensus.cc:1671] finished
0806 04:01:33.927754 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:34.090845 (+163091us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17216,"raft.queue_time_us":26165,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":7862528,"thread_start_us":155405,"threads_started":1,"wal-append.queue_time_us":333503}
I20260806 04:01:34.234218 4436 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5.
W20260806 04:01:34.130149 4515 log.cc:924] Time spent T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20260806 04:01:34.244263 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 829) took 1473 ms. Trace:
I20260806 04:01:34.244709 1192 rpcz_store.cc:276] 0806 04:01:32.771338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.781895 (+ 10557us) service_pool.cc:227] Handling call
0806 04:01:32.804249 (+ 22354us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:32.892631 (+ 88382us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:32.892653 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:32.893042 (+ 389us) log.cc:842] Serialized 26 byte log entry
0806 04:01:32.986097 (+ 93055us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:33.041352 (+ 55255us) log.cc:842] Serialized 14 byte log entry
0806 04:01:33.127184 (+ 85832us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:34.243670 (+1116486us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b54003faa54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:34.243722 (+ 52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:34.243776 (+ 54us) raft_consensus.cc:1671] finished
0806 04:01:34.244002 (+ 226us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:34.244191 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":124111,"prepare.queue_time_us":54723,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":174644,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":7742,"spinlock_wait_cycles":79955840,"thread_start_us":92795,"threads_started":1,"wal-append.queue_time_us":193896}
I20260806 04:01:33.866314 4325 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:34.184911 4323 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:34.304670 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.305398 1190 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:34.305723 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.306329 1187 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:34.308552 1187 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 7 LEARNER]: Advancing to term 8
I20260806 04:01:34.293844 4264 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:34.309454 4264 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.313006 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.315191 32614 raft_consensus.cc:1226] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260806 04:01:34.315739 32615 raft_consensus.cc:1226] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 04:01:34.318516 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.076809 4429 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.321324 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:34.322613 1190 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:34.323071 32614 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:34.325084 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.300940 4323 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:34.294034 4486 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.146768 4008 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.325877 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.332271 32615 raft_consensus.cc:1261] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:01:34.341480 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:34.342218 32603 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:34.318343 4200 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:34.369961 32614 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 04:01:34.370769 4399 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.583s user 0.008s sys 0.004s
I20260806 04:01:34.371012 4327 consensus_queue.cc:237] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:34.371244 4399 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:34.292721 4509 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:34.322170 4007 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.381202 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.374603 1187 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 8 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:34.375408 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:34.370644 4504 log.cc:924] Time spent T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
I20260806 04:01:34.388470 1187 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260806 04:01:34.396766 32745 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.022s sys 0.048s
W20260806 04:01:34.400038 32745 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s user 0.023s sys 0.048s
W20260806 04:01:34.407094 4529 log.cc:924] Time spent T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260806 04:01:34.407243 1184 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:34.407624 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.380853 4098 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:34.482337 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 743) took 2028 ms. Trace:
I20260806 04:01:34.486436 4325 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:34.373983 4200 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.365630 4486 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.495752 32475 rpcz_store.cc:276] 0806 04:01:32.453943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.464622 (+ 10679us) service_pool.cc:227] Handling call
0806 04:01:32.479952 (+ 15330us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:33.458630 (+978678us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:34.482088 (+1023458us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b540009e054. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:34.482265 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2249323904}
I20260806 04:01:34.503274 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 750) took 1032 ms. Trace:
I20260806 04:01:34.505239 4323 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:34.505612 4008 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:34.518908 32474 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.009s sys 0.029s
W20260806 04:01:34.519282 32474 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.568s user 0.009s sys 0.029s
I20260806 04:01:34.392542 4007 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.510970 1185 rpcz_store.cc:276] 0806 04:01:33.470859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.566169 (+ 95310us) service_pool.cc:227] Handling call
0806 04:01:33.566504 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:33.682664 (+116160us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:01:33.682698 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:33.693257 (+ 10559us) log.cc:842] Serialized 26 byte log entry
0806 04:01:33.846898 (+153641us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:33.846941 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:33.847015 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:34.503075 (+656060us) raft_consensus.cc:1671] finished
0806 04:01:34.503097 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:34.503216 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56870,"raft.queue_time_us":73332,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":39542,"thread_start_us":62414,"threads_started":1,"wal-append.queue_time_us":171770}
W20260806 04:01:34.517287 1137 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.027s sys 0.077s
I20260806 04:01:34.507790 4200 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:34.525929 1137 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.639s user 0.027s sys 0.077s
W20260806 04:01:34.527024 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 639843us
W20260806 04:01:34.522357 4421 consensus_meta.cc:287] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.678s user 0.004s sys 0.007s
I20260806 04:01:34.525000 4399 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:34.535820 4421 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:34.571772 32741 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 04:01:34.569492 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 376293us
I20260806 04:01:34.576375 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.592231 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.592654 1137 leader_election.cc:398] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4.
I20260806 04:01:34.592970 1185 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:34.593050 1137 leader_election.cc:401] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:34.600839 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.605540 4421 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.606151 4421 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.607528 4414 raft_consensus.cc:3032] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:34.608135 4414 raft_consensus.cc:729] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:34.608230 4266 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4.
I20260806 04:01:34.609150 4414 consensus_queue.cc:260] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:34.610363 4007 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:34.610775 4414 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:34.611251 4421 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:34.613562 32746 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.016s sys 0.075s
W20260806 04:01:34.619082 32746 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s user 0.016s sys 0.075s
I20260806 04:01:34.619217 32427 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:34.628197 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.015s sys 0.020s
W20260806 04:01:34.628688 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.517s user 0.015s sys 0.020s
I20260806 04:01:34.612100 4486 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:34.627744 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:34.630852 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:34.631968 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:34.625912 4421 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:34.628759 4378 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.798s user 0.013s sys 0.018s
W20260806 04:01:34.639956 3647 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.022s sys 0.029s
I20260806 04:01:34.634873 3261 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:34.653260 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 270488us
I20260806 04:01:34.638614 4266 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:34.660411 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:34.663867 32551 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:34.639030 3697 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:34.664513 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:34.665874 32478 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:34.666054 32551 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:34.666280 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:34.673585 32551 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:34.655575 4378 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.826s user 0.014s sys 0.021s
W20260806 04:01:34.675071 32682 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.013s sys 0.010s
W20260806 04:01:34.694753 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 62881.6us
I20260806 04:01:34.695690 32427 leader_election.cc:398] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4.
I20260806 04:01:34.696712 32427 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:34.697896 32427 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:34.691642 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:34.692725 32682 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.533s user 0.013s sys 0.012s
I20260806 04:01:34.693702 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:34.701327 32474 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:34.702258 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 543975us
I20260806 04:01:34.782683 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:34.785120 32604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1904) took 1237 ms (client timeout 1544 ms). Trace:
W20260806 04:01:34.789753 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 89280.1us
W20260806 04:01:34.656881 3647 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.023s sys 0.030s
W20260806 04:01:34.786319 32604 rpcz_store.cc:269] 0806 04:01:33.547658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.550654 (+ 2996us) service_pool.cc:227] Handling call
0806 04:01:34.784945 (+1234291us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b540035c354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:34.785055 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":350054784}
I20260806 04:01:34.630656 4530 consensus_queue.cc:1046] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260806 04:01:34.839651 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:34.842454 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 52027.7us
I20260806 04:01:34.844556 1182 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:34.665285 4339 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.854542 32474 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:34.844974 4266 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.007s
W20260806 04:01:34.846753 32688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.026s sys 0.019s
I20260806 04:01:34.668119 3697 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.690663 4466 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:34.845706 32608 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1890) took 2199 ms (client timeout 1515 ms). Trace:
W20260806 04:01:34.870025 32608 rpcz_store.cc:269] 0806 04:01:32.646490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.647399 (+ 909us) service_pool.cc:227] Handling call
0806 04:01:34.845487 (+2198088us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b540034f3d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:34.845613 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1912184064}
I20260806 04:01:34.847301 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.857055 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:34.857450 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:34.872560 32604 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:34.857725 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:34.858145 1137 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:34.652551 4421 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.873324 32749 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:34.699477 4488 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:34.688181 4200 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:34.693696 4098 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.152s
I20260806 04:01:34.690320 4008 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:34.864286 32688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.855s user 0.027s sys 0.019s
I20260806 04:01:34.784704 4326 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:34.898259 1137 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:34.785418 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:34.904619 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:34.895942 32682 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:34.911906 32604 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:34.698783 4273 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.181s
I20260806 04:01:34.861385 4546 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:34.878024 4488 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:34.919736 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 136123us
I20260806 04:01:34.878690 4000 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.885895 4323 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:34.687597 4542 consensus_queue.cc:1046] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.238s
I20260806 04:01:34.869072 4328 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.868122 3697 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:34.885031 4200 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:34.837513 4511 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.222s
I20260806 04:01:34.898057 3261 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:34.892198 4430 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:34.931488 4488 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:34.921892 32430 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.010s sys 0.022s
I20260806 04:01:34.929332 32603 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:34.956766 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:34.955826 32430 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.844s user 0.011s sys 0.022s
I20260806 04:01:34.950878 32615 raft_consensus.cc:1226] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:34.867818 4339 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:34.950264 32610 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1892) took 2268 ms (client timeout 2190 ms). Trace:
I20260806 04:01:34.924568 3647 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:34.961426 32466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59514 (request call id 1953) took 1291 ms (client timeout 1720 ms). Trace:
W20260806 04:01:34.959985 32610 rpcz_store.cc:269] 0806 04:01:32.681856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.682009 (+ 153us) service_pool.cc:227] Handling call
0806 04:01:34.950063 (+2268054us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540034e4d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:34.950189 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2679452544}
I20260806 04:01:34.965229 32748 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:34.965875 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2284) took 1433 ms. Trace:
W20260806 04:01:34.961833 32466 rpcz_store.cc:269] 0806 04:01:33.658650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.658875 (+ 225us) service_pool.cc:227] Handling call
0806 04:01:34.949720 (+1290845us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54003692d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:34.949850 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":540839296}
I20260806 04:01:34.935259 4003 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:34.968462 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:34.966168 32748 rpcz_store.cc:276] 0806 04:01:33.532013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.532976 (+ 963us) service_pool.cc:227] Handling call
0806 04:01:34.954440 (+1421464us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b5800004298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:34.954651 (+ 211us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:34.965630 (+ 10979us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:34.965823 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3011968}
I20260806 04:01:34.996104 32610 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:34.933764 4042 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:34.930627 4357 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:34.937700 4200 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:34.900730 4466 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.924171 4326 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260806 04:01:35.001400 4003 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:35.012310 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:34.962131 4266 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:35.039891 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 336802us
I20260806 04:01:35.010891 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:34.964919 3261 raft_consensus.cc:686] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:34.935277 4328 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.936615 4000 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:34.928414 4540 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260806 04:01:35.049182 4466 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.004369 4258 raft_consensus.cc:3032] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 04:01:35.028791 4381 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.924s user 0.012s sys 0.000s
I20260806 04:01:34.961205 4430 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:35.040468 4003 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:35.124125 32610 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:35.054572 4266 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:35.134270 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 90297.1us
I20260806 04:01:35.135025 32749 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
I20260806 04:01:35.137157 4323 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:35.137764 4357 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.138542 4357 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.143112 4541 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:35.143975 4541 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.144780 4339 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.124135 4008 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.146139 4008 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.149088 4357 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:35.149524 4543 log.cc:924] Time spent T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
W20260806 04:01:35.151432 4545 log.cc:924] Time spent T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.292s user 0.001s sys 0.000s
I20260806 04:01:35.153355 4421 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.154587 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:35.162588 4008 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:34.994930 3647 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.220552 4381 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 04:01:35.222457 4495 log.cc:924] Time spent T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 1.606s user 0.001s sys 0.004s
I20260806 04:01:35.224959 4438 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:35.225790 4438 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.229779 4360 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:35.230082 4546 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:35.234848 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1659) took 2789 ms. Trace:
I20260806 04:01:35.235306 32467 rpcz_store.cc:276] 0806 04:01:32.445753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.445898 (+ 145us) service_pool.cc:227] Handling call
0806 04:01:32.446208 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:33.341572 (+895364us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:35.234527 (+1892955us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54002021d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:35.234724 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2020886912}
I20260806 04:01:35.237162 4418 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:35.239027 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 340850us
I20260806 04:01:35.231086 4546 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:35.184525 4533 log.cc:924] Time spent T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260806 04:01:35.239868 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:35.121820 4258 raft_consensus.cc:729] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:35.182861 4328 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.266882 4258 consensus_queue.cc:260] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.011806 4042 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.271577 4258 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:35.275244 3647 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.271775 4357 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:35.275887 4541 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.277053 4357 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:35.277575 4357 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:35.278576 4546 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:35.282402 4042 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.185130 3261 consensus_queue.cc:237] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.186852 4000 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.289106 32746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 511) took 1624 ms. Trace:
I20260806 04:01:35.290388 32746 rpcz_store.cc:276] 0806 04:01:33.664705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.698384 (+ 33679us) service_pool.cc:227] Handling call
0806 04:01:33.698786 (+ 402us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:33.851874 (+153088us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:33.851907 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:33.866457 (+ 14550us) log.cc:842] Serialized 52 byte log entry
0806 04:01:34.629213 (+762756us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:34.975313 (+346100us) log.cc:842] Serialized 14 byte log entry
0806 04:01:34.975530 (+ 217us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:34.975600 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:35.288874 (+313274us) raft_consensus.cc:1671] finished
0806 04:01:35.288894 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:35.289046 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155079,"prepare.queue_time_us":543457,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":410544,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":21664,"thread_start_us":1065909,"threads_started":2,"wal-append.queue_time_us":1116475}
W20260806 04:01:35.292424 4494 log.cc:924] Time spent T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 1.487s user 0.037s sys 0.047s
I20260806 04:01:35.238160 4418 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.180035 4430 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:35.241003 4266 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:35.285981 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 46209.6us
I20260806 04:01:35.223749 4381 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260806 04:01:35.307390 4438 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.310417 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:35.310705 4532 log.cc:924] Time spent T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.709s user 0.003s sys 0.000s
I20260806 04:01:35.307991 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:35.312435 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2277) took 2158 ms. Trace:
I20260806 04:01:35.312856 32745 rpcz_store.cc:276] 0806 04:01:33.154267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.155072 (+ 805us) service_pool.cc:227] Handling call
0806 04:01:33.170514 (+ 15442us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:33.201349 (+ 30835us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:33.201382 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:33.804956 (+603574us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b180003af00. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a2f9f96 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a
0806 04:01:33.805627 (+ 671us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b2c00108868. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:33.829048 (+ 23421us) log.cc:842] Serialized 26 byte log entry
0806 04:01:34.492847 (+663799us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:34.492885 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:34.492954 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:35.309538 (+816584us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:35.312185 (+ 2647us) raft_consensus.cc:1671] finished
0806 04:01:35.312206 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:35.312333 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26967,"raft.queue_time_us":30533,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":15571,"spinlock_wait_cycles":54178432,"thread_start_us":606036,"threads_started":1,"wal-append.queue_time_us":706778}
I20260806 04:01:35.311826 1137 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:35.323992 32610 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:35.311973 4307 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:35.343561 4105 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.363988 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1954) took 1703 ms. Trace:
I20260806 04:01:35.367120 32477 rpcz_store.cc:276] 0806 04:01:33.660847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.660969 (+ 122us) service_pool.cc:227] Handling call
0806 04:01:33.696425 (+ 35456us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:34.320561 (+624136us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:34.320596 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:34.320640 (+ 44us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:34.320671 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:35.363712 (+1043041us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b540026fa54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:35.363758 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:35.363922 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1434860416}
W20260806 04:01:35.346009 4414 consensus_meta.cc:287] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.735s user 0.010s sys 0.006s
I20260806 04:01:35.374469 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2283) took 1846 ms. Trace:
I20260806 04:01:35.335749 4432 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:35.374864 32741 rpcz_store.cc:276] 0806 04:01:33.528192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.528308 (+ 116us) service_pool.cc:227] Handling call
0806 04:01:33.783061 (+254753us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b5800004298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:33.783308 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:34.572639 (+789331us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:35.374228 (+801589us) spinlock_profiling.cc:243] Waited 788 ms on lock 0x7b5400245254. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:35.374408 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1820191488}
W20260806 04:01:35.376964 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.722s user 0.019s sys 0.022s
W20260806 04:01:35.377319 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.723s user 0.019s sys 0.022s
I20260806 04:01:35.357754 4105 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.368750 4307 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.402354 4418 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:35.409987 32553 connection.cc:232] RPC call timeout handler was delayed by -1.7051610189955682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20590000000000003s for a response
I20260806 04:01:35.414762 4488 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:35.429637 32553 connection.cc:232] RPC call timeout handler was delayed by -1.1696609128266573s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19190000000000002s for a response
W20260806 04:01:35.433370 1136 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.012s sys 0.044s
W20260806 04:01:35.435143 1136 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.536s user 0.012s sys 0.044s
I20260806 04:01:35.418525 4007 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.435313 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:35.408631 4307 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.471997 4509 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:35.429580 4426 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.431178 4098 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:35.500752 32682 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.027s sys 0.029s
W20260806 04:01:35.470201 32604 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.003s sys 0.050s
W20260806 04:01:35.511763 32604 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.572s user 0.003s sys 0.051s
I20260806 04:01:35.496042 4381 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.482939 4565 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.495198 32466 raft_consensus.cc:1261] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:01:35.473578 4344 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:35.506151 32469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.023s sys 0.034s
W20260806 04:01:35.552285 32469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.551s user 0.023s sys 0.034s
I20260806 04:01:35.506618 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:35.552549 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.030s sys 0.036s
I20260806 04:01:35.433403 4008 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:35.552927 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.550s user 0.030s sys 0.036s
I20260806 04:01:35.437062 3697 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.431012 4328 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:35.471045 32688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.018s sys 0.036s
I20260806 04:01:35.515158 4396 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:01:35.514387 32682 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.588s user 0.027s sys 0.029s
I20260806 04:01:35.517774 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:35.561928 32688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.639s user 0.019s sys 0.036s
I20260806 04:01:35.547435 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2085) took 2672 ms. Trace:
I20260806 04:01:35.498791 4105 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:35.561641 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:35.511652 4156 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.013s sys 0.021s
I20260806 04:01:35.549474 32466 raft_consensus.cc:1261] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20260806 04:01:35.509929 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 74695.7us
I20260806 04:01:35.517251 4098 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:35.518815 4381 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.511022 4007 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:35.607503 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:35.512615 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:35.566720 32609 rpcz_store.cc:276] 0806 04:01:32.844861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.844973 (+ 112us) service_pool.cc:227] Handling call
0806 04:01:32.845315 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:33.003799 (+158484us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:33.034754 (+ 30955us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:33.034806 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:33.034848 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:35.516851 (+2482003us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54003340d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:35.516895 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:35.517021 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":364058240}
W20260806 04:01:35.568876 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 672965us
W20260806 04:01:35.560781 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 562031us
I20260806 04:01:35.607203 32741 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260806 04:01:35.592397 32683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.013s sys 0.002s
I20260806 04:01:35.624200 32466 pending_rounds.cc:76] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 2
I20260806 04:01:35.637338 32466 pending_rounds.cc:97] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 04:01:35.625710 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:35.577570 32478 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:35.637688 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:35.574100 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:35.638465 32748 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
W20260806 04:01:35.639020 32683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.014s sys 0.003s
I20260806 04:01:35.625473 32475 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:35.641161 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:35.551369 1184 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.006s sys 0.007s
I20260806 04:01:35.642221 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:35.637704 32466 raft_consensus.cc:2866] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:35.514771 4426 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:35.640630 32610 raft_consensus.cc:2379] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:35.641276 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 682643us
I20260806 04:01:35.636324 32553 leader_election.cc:302] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:35.650619 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:35.643079 32748 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:35.558291 3800 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:35.514642 4509 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.648352 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:35.565500 3790 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:35.645630 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:35.662639 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:35.663232 32471 raft_consensus.cc:2379] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:35.619530 4266 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:35.616289 4561 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.565755 4396 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:35.611717 4482 log.cc:924] Time spent T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 2.038s user 0.000s sys 0.008s
I20260806 04:01:35.650487 32428 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:35.703377 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 53015.7us
I20260806 04:01:35.654055 1137 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:35.617173 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:35.664446 32475 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260806 04:01:35.706797 32475 pending_rounds.cc:76] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 5
W20260806 04:01:35.706763 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 52790.7us
I20260806 04:01:35.707261 32475 pending_rounds.cc:97] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260806 04:01:35.707592 32475 raft_consensus.cc:2866] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 04:01:35.644249 1184 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.676s user 0.006s sys 0.007s
W20260806 04:01:35.624577 4156 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s user 0.013s sys 0.026s
I20260806 04:01:35.649170 4414 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:35.616253 4145 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:35.620544 4575 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:35.561046 4544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.006s sys 0.023s
W20260806 04:01:35.646504 4378 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.012s sys 0.006s
I20260806 04:01:35.611928 3697 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.553049 4565 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.687212 4430 consensus_queue.cc:1035] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:35.687461 4571 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.007s
I20260806 04:01:35.562718 4008 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:35.648970 4346 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.606745 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:35.687276 32749 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:35.553521 4344 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:35.616935 4488 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:35.618844 4307 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:35.659978 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:35.569854 4328 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:35.617901 4003 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.614933 4357 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.679059 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:36.730280 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1678) took 1094 ms. Trace:
I20260806 04:01:36.731463 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2303) took 1088 ms. Trace:
I20260806 04:01:36.732365 32745 rpcz_store.cc:276] 0806 04:01:35.642707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.677432 (+ 34725us) service_pool.cc:227] Handling call
0806 04:01:36.730936 (+1053504us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:36.731116 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:36.731140 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:36.731177 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:36.731210 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:36.731259 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:36.731408 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:36.241370 32429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.013s sys 0.023s
I20260806 04:01:35.653393 4418 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:36.734965 32749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2302) took 1095 ms. Trace:
I20260806 04:01:36.730700 32475 rpcz_store.cc:276] 0806 04:01:35.636303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.641595 (+ 5292us) service_pool.cc:227] Handling call
0806 04:01:35.664119 (+ 22524us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:36.730009 (+1065890us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:36.730216 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:35.747079 32614 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.014s sys 0.025s
I20260806 04:01:36.736372 1136 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:36.736548 32478 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
W20260806 04:01:36.733946 32429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.723s user 0.021s sys 0.025s
I20260806 04:01:36.743428 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1680) took 1088 ms. Trace:
I20260806 04:01:36.137326 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 759) took 1430 ms. Trace:
W20260806 04:01:36.291363 32609 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.017s sys 0.016s
W20260806 04:01:35.748764 32427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.011s sys 0.021s
W20260806 04:01:36.731738 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.0954e+06us
I20260806 04:01:36.745626 32553 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:36.729616 32615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.085s user 0.020s sys 0.022s
I20260806 04:01:36.744277 1185 rpcz_store.cc:276] 0806 04:01:34.706834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:34.706964 (+ 130us) service_pool.cc:227] Handling call
0806 04:01:34.707258 (+ 294us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:35.030415 (+323157us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:35.030454 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:35.070539 (+ 40085us) log.cc:842] Serialized 26 byte log entry
0806 04:01:35.551620 (+481081us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:35.585286 (+ 33666us) log.cc:842] Serialized 14 byte log entry
0806 04:01:35.585563 (+ 277us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:35.585645 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:36.137055 (+551410us) raft_consensus.cc:1671] finished
0806 04:01:36.137080 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:36.137248 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":158910,"prepare.queue_time_us":16449,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":137632,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":187,"thread_start_us":609403,"threads_started":2,"wal-append.queue_time_us":1003069}
I20260806 04:01:36.741762 32749 rpcz_store.cc:276] 0806 04:01:35.639654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.649253 (+ 9599us) service_pool.cc:227] Handling call
0806 04:01:35.686946 (+ 37693us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:36.734733 (+1047787us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:36.734913 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:36.744992 32427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.588s user 0.037s sys 0.062s
W20260806 04:01:36.746325 32615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.102s user 0.020s sys 0.022s
W20260806 04:01:36.744567 32609 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.043s user 0.020s sys 0.021s
W20260806 04:01:36.742517 32614 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.781s user 0.020s sys 0.055s
I20260806 04:01:36.730515 32682 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:36.750280 32427 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:36.754721 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 836) took 3750 ms. Trace:
I20260806 04:01:36.743883 32478 rpcz_store.cc:276] 0806 04:01:35.654980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.659733 (+ 4753us) service_pool.cc:227] Handling call
0806 04:01:36.743353 (+1083620us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:36.751665 4091 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260806 04:01:36.759613 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:36.760674 4008 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:36.751819 4139 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:35.731225 4092 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:36.761448 4092 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:36.761368 4139 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:36.763546 4139 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:35.701501 4006 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.217s user 0.021s sys 0.020s
I20260806 04:01:36.751833 32737 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
W20260806 04:01:36.737563 4378 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.851s user 0.014s sys 0.024s
I20260806 04:01:36.759130 4582 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:36.738247 4544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.799s user 0.007s sys 0.041s
I20260806 04:01:36.761879 4092 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:36.571398 4492 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.953s user 0.021s sys 0.053s
I20260806 04:01:36.755540 1188 rpcz_store.cc:276] 0806 04:01:32.987959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:32.988111 (+ 152us) service_pool.cc:227] Handling call
0806 04:01:32.988469 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:33.127050 (+138581us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:33.127082 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:33.140347 (+ 13265us) log.cc:842] Serialized 26 byte log entry
0806 04:01:33.407539 (+267192us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:33.407575 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:33.407640 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:35.611119 (+2203479us) spinlock_profiling.cc:243] Waited 57.1 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:36.734549 (+1123430us) raft_consensus.cc:1671] finished
0806 04:01:36.734569 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:36.737958 (+ 3389us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":141205,"raft.queue_time_us":121240,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":15348,"spinlock_wait_cycles":131467904,"thread_start_us":236589,"threads_started":1,"wal-append.queue_time_us":372926}
I20260806 04:01:36.767802 1136 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:36.749888 3790 consensus_meta.cc:287] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 1.069s user 0.008s sys 0.004s
I20260806 04:01:35.683322 4327 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:36.769964 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:35.713359 4581 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:36.535449 4189 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.701485 4396 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:36.774080 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:35.972441 4538 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:36.240974 3261 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.972112 4539 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:36.503618 4426 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:36.778635 32471 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:36.447501 4381 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:36.777050 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2304) took 1096 ms. Trace:
I20260806 04:01:35.679396 3800 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:36.540723 4509 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.009763 4581 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:35.900743 32740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2289) took 1252 ms (client timeout 1541 ms). Trace:
W20260806 04:01:37.019078 32740 rpcz_store.cc:269] 0806 04:01:34.640783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:34.648637 (+ 7854us) service_pool.cc:227] Handling call
0806 04:01:35.616987 (+968350us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b5800004298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:35.893270 (+276283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3375488}
W20260806 04:01:36.766579 4006 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.284s user 0.052s sys 0.053s
I20260806 04:01:35.853582 4520 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:36.732069 4098 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:35.853307 4273 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:36.458127 4007 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:36.772920 4008 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:36.758615 4200 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.743s user 0.013s sys 0.009s
I20260806 04:01:36.739133 4346 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:36.764056 4139 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:36.752480 4418 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:36.767616 4307 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:36.764750 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33182 (request call id 527) took 1159 ms. Trace:
W20260806 04:01:36.994338 32477 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.113s user 0.076s sys 0.071s
I20260806 04:01:36.994644 32748 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 9: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:36.766306 4430 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:36.766467 4003 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:36.770329 4359 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:36.779958 4105 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:36.768258 4558 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:36.788746 32467 raft_consensus.cc:1261] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 04:01:37.079461 32467 pending_rounds.cc:76] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 2
I20260806 04:01:37.079756 32467 pending_rounds.cc:97] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 04:01:37.080051 32467 raft_consensus.cc:2866] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:37.079505 4558 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:36.777808 4492 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.159s user 0.021s sys 0.055s
I20260806 04:01:37.035157 4373 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:36.772724 4156 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:36.780308 4323 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:36.773058 3697 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:36.779848 1183 raft_consensus.cc:1226] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:37.009934 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.017834 32743 rpcz_store.cc:276] 0806 04:01:35.681163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:36.747440 (+1066277us) service_pool.cc:227] Handling call
0806 04:01:36.776566 (+ 29126us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:36.776741 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:36.776765 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:36.776816 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:36.776847 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:36.776900 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:36.776995 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:36.782096 4544 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:36.783946 4585 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:35.705802 4561 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:36.758237 4414 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.001124 4327 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:36.764232 4575 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:37.061779 32558 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.024s sys 0.041s
W20260806 04:01:37.106911 32558 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.213s user 0.025s sys 0.041s
I20260806 04:01:37.023826 3790 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:36.776708 4582 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:37.071079 32477 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.189s user 0.082s sys 0.080s
I20260806 04:01:37.072337 32470 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:37.078164 4430 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:37.079319 4105 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:35.745760 4400 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 1.045s
I20260806 04:01:36.766419 4357 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.050907 4139 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.011268 3261 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:37.088001 32746 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.479s user 0.126s sys 0.154s
I20260806 04:01:37.001968 4189 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:37.081059 4402 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:37.104791 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 510) took 3440 ms. Trace:
I20260806 04:01:37.122877 32739 rpcz_store.cc:276] 0806 04:01:33.664015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.664174 (+ 159us) service_pool.cc:227] Handling call
0806 04:01:33.664515 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:33.888282 (+223767us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:33.888313 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:33.888352 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:33.888380 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.104528 (+3216148us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540011d0d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:37.104577 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.104726 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514118016}
I20260806 04:01:37.050498 32737 rpcz_store.cc:276] 0806 04:01:35.605574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.606435 (+ 861us) service_pool.cc:227] Handling call
0806 04:01:36.764685 (+1158250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:37.042569 4273 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:37.029816 4539 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.044281 4008 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.113499 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1960) took 1622 ms. Trace:
I20260806 04:01:37.077488 4307 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.044293 4200 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.076074 32471 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:37.039671 4520 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:37.118932 32746 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.510s user 0.126s sys 0.154s
I20260806 04:01:37.132474 32746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 524) took 1527 ms. Trace:
I20260806 04:01:37.133455 4264 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.132839 32746 rpcz_store.cc:276] 0806 04:01:35.604691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.608347 (+ 3656us) service_pool.cc:227] Handling call
0806 04:01:35.608643 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:35.608843 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:01:35.609162 (+ 319us) log.cc:842] Serialized 14 byte log entry
0806 04:01:37.132067 (+1522905us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:37.132153 (+ 86us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:37.132213 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.132281 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.132409 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":178,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"spinlock_wait_cycles":1792,"thread_start_us":1522497,"threads_started":1,"wal-append.queue_time_us":1505403}
I20260806 04:01:37.091480 4323 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:37.105818 4327 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:37.139058 4327 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:37.123790 4600 log.cc:924] Time spent T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260806 04:01:37.088315 4092 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.121207 4530 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:37.113338 4430 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:37.128789 32477 rpcz_store.cc:276] 0806 04:01:35.491575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.491772 (+ 197us) service_pool.cc:227] Handling call
0806 04:01:35.498089 (+ 6317us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:35.498303 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:01:35.881364 (+383061us) log.cc:842] Serialized 14 byte log entry
0806 04:01:37.109101 (+1227737us) log.cc:842] Serialized 14 byte log entry
0806 04:01:37.113086 (+ 3985us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:37.113139 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:37.113236 (+ 97us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.113295 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.113433 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1545537,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":2006,"raft.queue_time_us":7338,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":5796,"spinlock_wait_cycles":9592064,"thread_start_us":1597788,"threads_started":2,"wal-append.queue_time_us":1221196}
I20260806 04:01:37.091929 3697 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.069664 4581 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:37.090758 4156 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.026856 4538 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.107576 4544 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.156014 4182 raft_consensus.cc:3032] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 04:01:37.159614 4182 raft_consensus.cc:729] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:37.121745 4328 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.161403 4328 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:37.135479 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 758) took 2429 ms. Trace:
I20260806 04:01:37.135941 4523 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:37.139583 4006 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:37.128537 4402 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:37.169448 32478 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:37.136642 4565 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.152629 4139 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.161759 4016 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:37.172230 4575 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:37.153766 4608 log.cc:924] Time spent T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260806 04:01:37.172734 4182 consensus_queue.cc:260] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.173600 4539 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.173780 4182 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:37.169832 4523 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.176245 4523 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.177453 4346 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.141579 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.157433 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1961) took 1665 ms. Trace:
I20260806 04:01:37.141626 4617 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:37.181269 32467 rpcz_store.cc:276] 0806 04:01:35.492286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.492407 (+ 121us) service_pool.cc:227] Handling call
0806 04:01:35.492721 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:37.080538 (+1587817us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.157192 (+ 76654us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540019a354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:37.157371 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2979334400}
I20260806 04:01:37.166821 4328 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.182011 4617 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:37.182518 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.160813 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.142822 4264 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:37.168426 32470 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:37.138705 4344 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.171891 4016 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.107331 4585 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:37.182543 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1964) took 1668 ms. Trace:
I20260806 04:01:37.238174 32474 rpcz_store.cc:276] 0806 04:01:35.513715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.513856 (+ 141us) service_pool.cc:227] Handling call
0806 04:01:35.514162 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:35.705336 (+191174us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:35.705369 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:36.731334 (+1025965us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:36.731377 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.182309 (+450932us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54002026d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:37.182355 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.182485 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":439159680}
W20260806 04:01:37.239389 4620 log.cc:924] Time spent T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
W20260806 04:01:37.240283 4606 log.cc:924] Time spent T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260806 04:01:37.180540 4006 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.241645 4520 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.242468 4006 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.183771 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.162503 4558 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.145525 4530 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:37.244560 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 496232us
W20260806 04:01:37.250111 32609 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s user 0.004s sys 0.006s
I20260806 04:01:37.190373 4156 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.234989 4617 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.234313 4418 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.235118 4105 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.237159 4323 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.253229 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 646) took 1749 ms. Trace:
I20260806 04:01:37.263636 32467 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:01:37.258260 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1681) took 1578 ms. Trace:
I20260806 04:01:37.237493 4264 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:37.515465 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:37.262179 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 377354us
W20260806 04:01:37.264464 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 514245us
I20260806 04:01:37.544494 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:37.536975 32694 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.001s sys 0.031s
I20260806 04:01:37.538962 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1959) took 3684 ms. Trace:
I20260806 04:01:37.581995 32743 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
W20260806 04:01:37.375447 32682 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.005s sys 0.007s
I20260806 04:01:37.583220 32427 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
W20260806 04:01:37.583765 32694 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.836s user 0.003s sys 0.032s
I20260806 04:01:37.544941 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:37.585593 32748 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:37.264580 32683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.009s sys 0.014s
W20260806 04:01:37.587743 32683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.840s user 0.031s sys 0.066s
I20260806 04:01:37.306494 32605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 632) took 2616 ms. Trace:
I20260806 04:01:37.540787 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:37.580858 32474 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
W20260806 04:01:37.588315 32682 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.835s user 0.006s sys 0.012s
W20260806 04:01:37.597517 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 867014us
I20260806 04:01:37.506769 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.599046 32471 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:37.573009 32469 rpcz_store.cc:276] 0806 04:01:35.680139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:36.747314 (+1067175us) service_pool.cc:227] Handling call
0806 04:01:36.752430 (+ 5116us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:37.257918 (+505488us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:37.257950 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:37.257990 (+ 40us) raft_consensus.cc:1631] Marking committed up to 10
0806 04:01:37.258020 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.258084 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.258194 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18558,"raft.queue_time_us":145125,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":4193,"thread_start_us":486494,"threads_started":1}
W20260806 04:01:37.259513 4510 log.cc:924] Time spent T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 2.959s user 0.006s sys 0.021s
W20260806 04:01:37.474509 1136 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.005s sys 0.046s
W20260806 04:01:37.552762 4553 log.cc:924] Time spent T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 2.372s user 0.003s sys 0.032s
I20260806 04:01:37.540326 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:37.516220 3261 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:37.964828 4623 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.063262 4623 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:38.064204 1136 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.280s user 0.010s sys 0.059s
W20260806 04:01:38.064904 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 1.29706e+06us
I20260806 04:01:38.065485 4623 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:38.065716 1136 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:37.959566 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 634) took 3267 ms. Trace:
I20260806 04:01:38.066762 32682 leader_election.cc:398] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4.
I20260806 04:01:37.264896 4273 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:37.902724 4327 consensus_meta.cc:287] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.753s user 0.007s sys 0.000s
I20260806 04:01:38.066408 32602 rpcz_store.cc:276] 0806 04:01:34.691940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:34.692062 (+ 122us) service_pool.cc:227] Handling call
0806 04:01:34.692383 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:34.692721 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:34.692750 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:34.693211 (+ 461us) log.cc:842] Serialized 26 byte log entry
0806 04:01:37.179894 (+2486683us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b18002229a0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb411 00007fd16d2285cf
0806 04:01:37.249489 (+ 69595us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:37.249529 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.249599 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:37.959225 (+709626us) raft_consensus.cc:1671] finished
0806 04:01:37.959247 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.959490 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15131,"raft.queue_time_us":1535,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":8678,"spinlock_wait_cycles":3889536,"thread_start_us":69417,"threads_started":1,"wal-append.queue_time_us":3121363}
I20260806 04:01:37.582202 32473 rpcz_store.cc:276] 0806 04:01:33.854193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.854352 (+ 159us) service_pool.cc:227] Handling call
0806 04:01:33.887906 (+ 33554us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:34.176964 (+289058us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:01:34.176995 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:34.177485 (+ 490us) log.cc:842] Serialized 26 byte log entry
0806 04:01:34.667065 (+489580us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:34.667103 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.538305 (+2871202us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b540019ad54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:37.538391 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:37.538460 (+ 69us) raft_consensus.cc:1671] finished
0806 04:01:37.538492 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.538879 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16179,"raft.queue_time_us":45489,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":626478336,"thread_start_us":489292,"threads_started":1,"wal-append.queue_time_us":792057,"wal-append.run_cpu_time_us":3635,"wal-append.run_wall_time_us":1173620}
I20260806 04:01:37.589517 32605 rpcz_store.cc:276] 0806 04:01:34.690444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:34.690556 (+ 112us) service_pool.cc:227] Handling call
0806 04:01:34.837852 (+147296us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:35.004742 (+166890us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:35.004775 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:35.004831 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:35.004864 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.306215 (+2301351us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54000cb554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:37.306266 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.306428 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":383284992}
W20260806 04:01:38.080317 1189 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.296s user 0.004s sys 0.006s
W20260806 04:01:38.080646 1189 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.304s user 0.004s sys 0.006s
I20260806 04:01:38.070235 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.682226 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.067735 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:37.686760 32429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.004s sys 0.014s
I20260806 04:01:38.085981 32682 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
W20260806 04:01:38.072454 32615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.301s user 0.000s sys 0.010s
I20260806 04:01:38.069087 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.262578 4156 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:38.075469 32551 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.006s sys 0.004s
W20260806 04:01:38.092389 32551 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.947s user 0.006s sys 0.004s
W20260806 04:01:38.086244 32429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.315s user 0.012s sys 0.025s
I20260806 04:01:38.025368 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:37.908805 4430 consensus_meta.cc:287] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.761s user 0.008s sys 0.000s
I20260806 04:01:37.566296 4328 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.565444 4307 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.557956 4544 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.090623 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:37.703711 32748 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:38.072883 32609 rpcz_store.cc:276] 0806 04:01:35.503354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.649013 (+145659us) service_pool.cc:227] Handling call
0806 04:01:35.649288 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:35.649487 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:36.749345 (+1099858us) log.cc:842] Serialized 14 byte log entry
0806 04:01:37.252898 (+503553us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:37.252952 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:37.253003 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.253062 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.253174 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":485808,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":89,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":6838,"thread_start_us":1548751,"threads_started":2,"wal-append.queue_time_us":788279}
W20260806 04:01:38.076543 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 493375us
I20260806 04:01:38.093003 32477 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
W20260806 04:01:38.086835 32615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.315s user 0.000s sys 0.010s
W20260806 04:01:38.093104 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 947995us
I20260806 04:01:38.125018 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:37.960872 4625 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:37.911445 4396 consensus_meta.cc:287] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.902s user 0.008s sys 0.000s
I20260806 04:01:37.558971 4538 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.565001 4582 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.536746 3697 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:37.553102 4561 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:37.561025 4189 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:38.072310 4007 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.000s sys 0.009s
I20260806 04:01:37.603099 4421 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.118701 4439 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:37.907507 4435 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:37.562776 4585 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:37.604012 4105 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:38.127372 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2106) took 2971 ms. Trace:
I20260806 04:01:37.628072 4451 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:38.172586 4007 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s user 0.000s sys 0.012s
I20260806 04:01:38.129074 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.183329 32615 rpcz_store.cc:276] 0806 04:01:35.156316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.316022 (+159706us) service_pool.cc:227] Handling call
0806 04:01:35.316356 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:35.316560 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 04:01:35.644341 (+327781us) log.cc:842] Serialized 14 byte log entry
0806 04:01:38.127047 (+2482706us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:38.127101 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:38.127148 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:38.127208 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:38.127314 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22523,"prepare.queue_time_us":818409,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":295082,"raft.run_cpu_time_us":685,"raft.run_wall_time_us":28224,"thread_start_us":2766196,"threads_started":3,"wal-append.queue_time_us":1366795,"wal-append.run_cpu_time_us":1300,"wal-append.run_wall_time_us":1065155}
I20260806 04:01:37.681135 4636 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:01:38.169942 32551 leader_election.cc:334] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 2.059s (SENT)
W20260806 04:01:38.125115 4523 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.020s sys 0.018s
I20260806 04:01:38.092173 4632 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:38.170846 32427 connection.cc:232] RPC call timeout handler was delayed by -1.4205457232892513s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1724s for a response
I20260806 04:01:38.095013 4156 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.094966 4182 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:38.095867 4617 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:38.078836 4008 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.175143 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.094168 32737 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:38.174136 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:38.177793 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 89582.6us
I20260806 04:01:38.127768 4623 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:01:38.172048 4581 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.021s sys 0.062s
W20260806 04:01:38.181144 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 92449.6us
I20260806 04:01:38.209384 4623 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:38.209820 4623 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:38.210628 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.168036 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.211829 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.211937 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.213162 1186 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:01:38.212936 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.128991 32467 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:38.208966 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.088523 4327 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.129876 4328 consensus_queue.cc:1046] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260806 04:01:38.095216 4637 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.121950 4430 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.175292 4582 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.195833 4182 raft_consensus.cc:686] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:38.221468 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:38.238960 1186 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:38.188547 4636 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:38.186524 4105 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.185395 4435 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:38.198727 32470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1679) took 2551 ms (client timeout 3327 ms). Trace:
I20260806 04:01:38.199182 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.200153 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 870) took 5207 ms. Trace:
I20260806 04:01:38.200343 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2094) took 4689 ms. Trace:
I20260806 04:01:38.257846 32612 rpcz_store.cc:276] 0806 04:01:33.501297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.508905 (+ 7608us) service_pool.cc:227] Handling call
0806 04:01:33.509293 (+ 388us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:33.542843 (+ 33550us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:33.542874 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:33.543394 (+ 520us) log.cc:842] Serialized 26 byte log entry
0806 04:01:33.820152 (+276758us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:33.820194 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:36.731051 (+2910857us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b540007f3d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:36.731096 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:37.144423 (+413327us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:38.190424 (+1046001us) raft_consensus.cc:1671] finished
0806 04:01:38.190444 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:38.190565 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1922,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":7338,"spinlock_wait_cycles":85578496,"thread_start_us":276191,"threads_started":1,"wal-append.queue_time_us":378443}
I20260806 04:01:38.184002 4439 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.135304 4625 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.267176 4625 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.171296 4396 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:38.269176 4396 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.203466 4200 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.185233 4585 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.270421 4585 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.272339 4430 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.207858 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:38.208803 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 93614.3us
W20260806 04:01:38.272637 4653 log.cc:924] Time spent T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260806 04:01:38.273485 32746 raft_consensus.cc:2396] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260806 04:01:38.248031 4637 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.273818 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:38.274387 1185 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:38.275329 4637 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.217278 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.276389 32602 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:38.277374 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.277859 32738 raft_consensus.cc:1226] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:38.278507 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2300) took 2640 ms. Trace:
I20260806 04:01:38.278832 32738 rpcz_store.cc:276] 0806 04:01:35.638414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.648813 (+ 10399us) service_pool.cc:227] Handling call
0806 04:01:35.687786 (+ 38973us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:38.277694 (+2589908us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54002c7520. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d1e4730 00007fd16d256300 00007fd16d257371 00007fd16d253edf 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc
0806 04:01:38.278249 (+ 555us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:38.278439 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2481844608}
I20260806 04:01:38.213824 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.280048 32748 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:38.280889 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:38.281641 3800 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.171676 4538 consensus_queue.cc:1046] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260806 04:01:38.218853 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.212980 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:38.214051 4581 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.022s sys 0.062s
I20260806 04:01:38.219830 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.220054 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.218185 4617 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.220295 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.225143 32427 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:38.226310 32610 raft_consensus.cc:1261] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:01:38.228556 4266 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.228463 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.229152 3836 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:38.228747 4623 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.229715 4638 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.229813 4092 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.230306 4391 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.237794 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.188453 4451 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.221668 4327 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:38.241206 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 147127us
I20260806 04:01:38.193478 4264 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:38.249274 32470 rpcz_store.cc:269] 0806 04:01:35.647362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:35.647641 (+ 279us) service_pool.cc:227] Handling call
0806 04:01:38.198537 (+2550896us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400201cd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:38.198643 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":784983424}
I20260806 04:01:38.249656 1187 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:01:38.264045 4182 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.207621 1188 raft_consensus.cc:1261] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:38.213500 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:38.192140 4523 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.023s sys 0.020s
I20260806 04:01:38.267793 32683 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:38.177261 4421 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:38.237321 4656 consensus_queue.cc:1046] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
W20260806 04:01:38.399353 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 66993.1us
I20260806 04:01:38.399545 1190 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:38.400705 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.262742 32745 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:38.398010 32614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 635) took 3669 ms. Trace:
I20260806 04:01:38.345974 1185 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:38.402864 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1969) took 1315 ms. Trace:
W20260806 04:01:38.403012 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 177952us
I20260806 04:01:38.403301 32466 rpcz_store.cc:276] 0806 04:01:37.087240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:37.087400 (+ 160us) service_pool.cc:227] Handling call
0806 04:01:37.087739 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:37.117832 (+ 30093us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:37.117864 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:37.117904 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:37.117934 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:38.391912 (+1273978us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b540019cb54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:38.391967 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:38.402781 (+ 10814us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68709248}
I20260806 04:01:38.281885 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.402491 32609 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:38.405515 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.405818 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.404392 32682 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:38.402599 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 134888us
I20260806 04:01:38.402518 32605 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:37.165095 1184 rpcz_store.cc:276] 0806 04:01:34.705851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:34.784179 (+ 78328us) service_pool.cc:227] Handling call
0806 04:01:34.877851 (+ 93672us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:34.915817 (+ 37966us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 04:01:36.762210 (+1846393us) log.cc:842] Serialized 14 byte log entry
0806 04:01:37.107758 (+345548us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:37.112676 (+ 4918us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:37.115383 (+ 2707us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:37.120337 (+ 4954us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:37.128852 (+ 8515us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":211284,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":7041,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":5058,"thread_start_us":2101625,"threads_started":2,"wal-append.queue_time_us":367230}
I20260806 04:01:38.409435 32603 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 2.
I20260806 04:01:38.409458 1186 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:38.410432 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:38.411476 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.412426 4165 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:38.412910 32608 raft_consensus.cc:1261] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:38.402451 32614 rpcz_store.cc:276] 0806 04:01:34.728233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:34.838828 (+110595us) service_pool.cc:227] Handling call
0806 04:01:34.839200 (+ 372us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:34.900286 (+ 61086us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:34.900323 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:34.916617 (+ 16294us) log.cc:842] Serialized 26 byte log entry
0806 04:01:36.749431 (+1832814us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:36.749473 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:38.397557 (+1648084us) spinlock_profiling.cc:243] Waited 60.6 ms on lock 0x7b5400305f54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:38.397605 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:38.397659 (+ 54us) raft_consensus.cc:1671] finished
0806 04:01:38.397670 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:38.397938 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4905,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":6738,"spinlock_wait_cycles":139488000,"thread_start_us":1788031,"threads_started":1,"wal-append.queue_time_us":2118123,"wal-append.run_cpu_time_us":2122,"wal-append.run_wall_time_us":1097616}
I20260806 04:01:38.413967 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 637) took 3625 ms. Trace:
I20260806 04:01:38.413743 4266 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.414363 32608 rpcz_store.cc:276] 0806 04:01:34.788827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.319827 (+3531000us) service_pool.cc:227] Handling call
0806 04:01:38.412596 (+ 92769us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:38.413735 (+ 1139us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:38.413917 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:38.252851 1183 rpcz_store.cc:276] 0806 04:01:32.992520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:33.167444 (+174924us) service_pool.cc:227] Handling call
0806 04:01:35.233919 (+2066475us) spinlock_profiling.cc:243] Waited 1.74 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4bce 000000000034aea9 00007fd172120a30 00007fd1721e67b4 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:35.279887 (+ 45968us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:37.102484 (+1822597us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:38.199865 (+1097381us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54004efcd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:38.200050 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3423488128}
I20260806 04:01:38.415556 4538 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:38.416283 4538 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:38.416817 4402 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.416450 32553 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:38.388489 4562 log.cc:924] Time spent T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 3.256s user 0.013s sys 0.050s
I20260806 04:01:38.417752 4662 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:38.406665 1188 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:38.417135 4538 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:38.419150 32553 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:38.412040 1185 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:38.446111 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.447718 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:38.448596 1188 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 9: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:38.246819 4636 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:38.449028 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 44730.2us
I20260806 04:01:38.458094 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.449380 32683 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:38.307003 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.449591 32427 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:38.279991 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.419596 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.303480 4657 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:38.460054 32748 raft_consensus.cc:2454] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:38.460729 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.460129 32612 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:38.464542 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.464774 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.464906 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.468021 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.468781 1188 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:38.467113 1186 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:38.468703 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.471796 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:38.472923 32612 raft_consensus.cc:2354] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
W20260806 04:01:38.471856 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 56051.3us
I20260806 04:01:38.474902 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:38.472905 32608 raft_consensus.cc:1226] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:01:38.272663 4585 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.477289 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:38.470299 1185 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:38.390656 4414 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.476123 32743 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:38.408733 4637 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:38.480216 32683 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:38.483254 4648 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:38.471613 1182 raft_consensus.cc:2379] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 6: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:38.485162 32469 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260806 04:01:38.485188 32470 raft_consensus.cc:1226] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:38.447625 3800 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:38.250582 4582 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.418241 32740 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:38.457278 4156 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.467127 4430 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:38.524302 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 44188.9us
I20260806 04:01:38.524709 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.467528 4625 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:38.486385 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.569955 32467 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:38.570897 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:38.530128 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 52905.1us
I20260806 04:01:38.530198 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.572649 32427 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:38.538065 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.472328 4200 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:38.483937 4460 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.577661 32427 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:38.544896 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.578076 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.545255 4105 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.547281 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:38.579262 32428 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:38.547739 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 885) took 1812 ms. Trace:
I20260806 04:01:38.548025 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.548403 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.479947 4327 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:38.547906 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.580825 32615 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:38.581542 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.479311 3790 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:38.486127 4538 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:38.418839 3836 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.556548 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.487589 32553 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:38.559787 32469 pending_rounds.cc:76] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 7
I20260806 04:01:38.543897 4667 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:38.563294 4530 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.563395 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.564244 4435 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.574295 4625 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:38.485642 4018 consensus_queue.cc:1046] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260806 04:01:38.565856 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.630510 32473 raft_consensus.cc:2379] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:38.541126 4571 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:38.547839 4266 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.551836 32737 raft_consensus.cc:2379] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:38.467646 4632 consensus_queue.cc:1035] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.092s
I20260806 04:01:38.543358 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.574376 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.638242 32614 raft_consensus.cc:2379] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:38.551592 4647 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.583081 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:38.638985 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 62091.1us
I20260806 04:01:38.583899 32477 raft_consensus.cc:1226] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:38.579907 1189 rpcz_store.cc:276] 0806 04:01:36.735210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:36.735773 (+ 563us) service_pool.cc:227] Handling call
0806 04:01:36.759209 (+ 23436us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:38.081044 (+1321835us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:38.081138 (+ 94us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:38.094460 (+ 13322us) log.cc:842] Serialized 26 byte log entry
0806 04:01:38.442462 (+348002us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:38.442507 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:38.442591 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:38.547510 (+104919us) raft_consensus.cc:1671] finished
0806 04:01:38.547531 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:38.547669 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17204,"raft.queue_time_us":245029,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":16506,"thread_start_us":1569268,"threads_started":2,"wal-append.queue_time_us":347114}
I20260806 04:01:38.584187 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.640322 32604 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:38.584519 32603 raft_consensus.cc:2379] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:38.584813 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 126540us
I20260806 04:01:38.469832 4657 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:38.642222 4439 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.643361 4657 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:38.643710 4451 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.583482 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.644227 4421 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:38.645210 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.646265 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.647200 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.419023 4638 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.614610 32469 pending_rounds.cc:97] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260806 04:01:38.651072 32469 raft_consensus.cc:2866] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:38.651823 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.418892 4391 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:38.615214 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 127698us
I20260806 04:01:38.616977 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.568980 4659 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:38.584944 4648 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:38.582862 4327 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:38.582711 3790 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:38.617726 4637 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:38.618068 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.626659 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.627631 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.570320 3800 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:38.614035 4585 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:38.467844 4165 consensus_queue.cc:1035] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260806 04:01:38.471557 4523 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260806 04:01:38.630766 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.582970 4460 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.616350 4667 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:38.636952 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.586411 4656 consensus_queue.cc:1035] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260806 04:01:38.576488 4486 consensus_queue.cc:1035] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260806 04:01:38.583367 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.689855 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.690286 32467 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:38.680579 4264 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260806 04:01:38.690943 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.629787 4625 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:38.632445 4571 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:38.691624 4571 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:38.644059 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:38.648207 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.660733 4644 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:38.585819 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.652343 32604 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:38.650983 32427 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:38.655853 32683 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:38.656296 4648 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:38.657123 32612 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:38.688357 32740 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:38.658216 32615 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:01:38.688589 4182 consensus_queue.cc:1046] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260806 04:01:38.658474 32738 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:38.701313 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.701596 4182 raft_consensus.cc:3032] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:38.701844 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.702184 32428 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:38.703239 4391 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:38.705860 4523 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:38.706367 4523 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:38.693892 4266 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260806 04:01:38.710222 4644 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.712182 4632 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:38.712378 4644 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.719053 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:38.719422 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 68517us
I20260806 04:01:38.647730 4647 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.720427 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:38.721885 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:38.722329 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.723047 32427 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:38.723701 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.724370 32748 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:38.687985 4667 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:38.726518 4091 raft_consensus.cc:3032] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:38.726907 4091 raft_consensus.cc:729] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:38.659420 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:38.656697 32682 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:38.661969 32551 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:38.648984 4396 raft_consensus.cc:2781] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:38.662643 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:38.665130 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.640746 1183 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:38.654961 4623 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.663358 1187 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:38.654718 4617 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.655086 4327 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:38.655424 3790 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:38.649988 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.693704 1189 raft_consensus.cc:2354] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:38.660605 4637 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:38.658871 4402 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:38.655910 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.698010 32471 raft_consensus.cc:1226] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:38.705896 4638 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.927831 1182 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
W20260806 04:01:38.891607 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 229700us
I20260806 04:01:38.926383 32610 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:38.929625 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.746433 4657 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:38.724022 4182 raft_consensus.cc:729] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:38.724593 4648 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:38.746991 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 90393.7us
I20260806 04:01:38.900163 4468 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:38.898881 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.680146 4488 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260806 04:01:38.900687 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.902518 32469 raft_consensus.cc:2379] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:38.906308 4582 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:38.890492 4636 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:38.890158 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 234935us
I20260806 04:01:38.906489 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.931614 32615 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 04:01:38.957923 32478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1684) took 1954 ms (client timeout 2211 ms). Trace:
I20260806 04:01:38.930389 4637 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:38.933267 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.655541 4659 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.933789 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.933981 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.934990 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.937204 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.937309 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.896523 4523 leader_election.cc:288] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.897823 4632 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:38.984423 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.985620 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.986740 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.987532 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:38.988679 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.989706 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.990372 32477 raft_consensus.cc:2379] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:38.955335 3800 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.993393 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.993863 32748 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:38.995276 32683 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:38.995748 32474 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:38.943094 32551 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:38.906591 4560 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:38.945151 32602 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:01:38.903921 4326 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:38.663937 4007 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20260806 04:01:39.000335 4326 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:39.000706 4326 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:38.903770 4585 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:38.935361 4677 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:38.951447 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.001577 4677 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:39.001780 32473 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:38.951717 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.002074 4677 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:39.002473 32741 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:39.003372 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:38.823184 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 100182us
I20260806 04:01:38.954308 1137 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:39.004532 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.005272 32741 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
W20260806 04:01:39.005313 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 51079.8us
I20260806 04:01:38.956609 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.007861 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.933637 4538 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:39.013825 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:38.958283 32478 rpcz_store.cc:269] 0806 04:01:37.003299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:37.003521 (+ 222us) service_pool.cc:227] Handling call
0806 04:01:38.957751 (+1954230us) spinlock_profiling.cc:243] Waited 79.1 ms on lock 0x7b54001b1a54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:38.957869 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":181971328}
I20260806 04:01:38.944898 3836 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:38.957345 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:38.957682 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:38.970806 4623 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:38.947885 4657 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:38.971776 32614 raft_consensus.cc:2379] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 7: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:38.973104 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:38.976672 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:38.979173 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:39.028334 32605 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 2.
I20260806 04:01:38.996779 32683 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:38.997269 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 54252.8us
I20260806 04:01:39.028782 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.004513 4488 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:39.015172 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.035615 32478 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:39.015415 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:39.015941 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.016314 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:38.990315 4091 consensus_queue.cc:260] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:39.057137 4657 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:38.938540 4402 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.059109 4402 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.014635 4264 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:39.019177 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 61578.7us
W20260806 04:01:38.970404 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 268285us
W20260806 04:01:39.063349 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 106087us
I20260806 04:01:39.037302 32749 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:38.997908 4560 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:39.066495 32682 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:39.013876 4585 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:39.022434 4523 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:39.022818 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.090955 32741 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:39.023264 4016 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.023710 4636 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:39.035115 4451 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:01:38.980513 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.018055 4182 consensus_queue.cc:260] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:38.958755 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:39.028280 32604 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:39.022336 4460 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.032833 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.121201 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.121434 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:39.033443 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:39.028488 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:39.037432 1185 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:38.703920 4671 raft_consensus.cc:2932] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:39.020025 4266 consensus_queue.cc:1035] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.021s
I20260806 04:01:39.014173 4538 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:39.107704 4667 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.134668 4538 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:39.010180 4139 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:39.047605 4662 consensus_meta.cc:287] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.629s user 0.004s sys 0.003s
I20260806 04:01:39.049234 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.050455 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.051692 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1685) took 2032 ms. Trace:
I20260806 04:01:39.027529 4396 raft_consensus.cc:686] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:39.053722 4414 raft_consensus.cc:3032] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:39.054558 4326 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:39.034267 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:39.052174 4623 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.021404 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:39.004492 4675 consensus_queue.cc:1035] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260806 04:01:39.087929 32682 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:39.067409 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.093217 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.052161 4659 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:39.090178 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:39.096832 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:39.097141 1136 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:39.098311 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:38.972474 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.035360 4488 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:39.057374 4091 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:39.103976 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:39.089560 4523 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:39.110492 4182 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:39.110860 4677 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:39.111428 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.097560 4638 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.015s
I20260806 04:01:39.098052 4430 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:39.112816 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33182 (request call id 542) took 1025 ms. Trace:
I20260806 04:01:39.114751 4327 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.115180 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2305) took 1507 ms. Trace:
I20260806 04:01:39.097694 4585 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.116272 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.101202 4007 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260806 04:01:39.087965 4560 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:39.123116 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 94705.7us
I20260806 04:01:39.126806 4542 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:39.127295 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:39.127517 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.043529 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.055920 4648 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.088120 4625 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.762768 32748 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 9: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:39.665438 32322 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36880 (request call id 232) took 12722 ms (client timeout 15000 ms). Trace:
I20260806 04:01:39.243608 32467 raft_consensus.cc:2379] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 8: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:39.765235 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 668116us
I20260806 04:01:39.769650 1137 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:39.135547 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.770586 32738 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:39.462652 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.771382 1186 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:39.772240 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 767) took 1556 ms. Trace:
W20260806 04:01:39.454681 1192 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 849) took 1244 ms (client timeout 1629 ms). Trace:
I20260806 04:01:39.135735 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.240813 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1772) took 1107 ms. Trace:
I20260806 04:01:39.757546 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.774714 1188 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:39.757539 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:39.463524 32474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1692) took 1925 ms (client timeout 1528 ms). Trace:
I20260806 04:01:39.765758 32477 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:39.762336 32743 rpcz_store.cc:276] 0806 04:01:38.087560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.087785 (+ 225us) service_pool.cc:227] Handling call
0806 04:01:39.112643 (+1024858us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54000a73d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.112758 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":805099392}
I20260806 04:01:39.771601 1136 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:39.460819 1187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 869) took 1237 ms (client timeout 1572 ms). Trace:
W20260806 04:01:39.766697 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 676528us
W20260806 04:01:39.316918 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 253798us
I20260806 04:01:39.766726 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55048 (request call id 792) took 1044 ms. Trace:
I20260806 04:01:39.775880 32614 raft_consensus.cc:1226] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:39.773622 1137 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:39.780345 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55048 (request call id 789) took 1143 ms. Trace:
I20260806 04:01:39.781829 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2319) took 1367 ms. Trace:
I20260806 04:01:39.781586 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:39.734400 3790 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.795s user 0.005s sys 0.005s
I20260806 04:01:39.818714 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.775543 1186 rpcz_store.cc:276] 0806 04:01:38.215410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.135230 (+919820us) service_pool.cc:227] Handling call
0806 04:01:39.772167 (+636937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.775125 32745 rpcz_store.cc:276] 0806 04:01:38.133036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.133151 (+ 115us) service_pool.cc:227] Handling call
0806 04:01:39.240589 (+1107438us) spinlock_profiling.cc:243] Waited 69 ms on lock 0x7b54000fa354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.240745 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158769408}
I20260806 04:01:39.816736 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:39.766081 32739 rpcz_store.cc:276] 0806 04:01:37.607620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.232354 (+624734us) service_pool.cc:227] Handling call
0806 04:01:39.113423 (+881069us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.113632 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:39.113655 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:39.113696 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:39.113722 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.113769 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:39.115008 (+ 1239us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b54000a73d0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.115124 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2025458048}
I20260806 04:01:39.817257 32738 rpcz_store.cc:276] 0806 04:01:38.414563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.955757 (+541194us) service_pool.cc:227] Handling call
0806 04:01:39.781656 (+825899us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54001f2a54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.781771 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1707746688}
I20260806 04:01:39.811116 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59514 (request call id 1978) took 1111 ms. Trace:
I20260806 04:01:39.832408 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.816112 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55048 (request call id 790) took 1160 ms. Trace:
I20260806 04:01:39.857120 32748 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
W20260806 04:01:39.853825 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 82289us
W20260806 04:01:39.817345 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 43819.5us
I20260806 04:01:39.816900 32467 rpcz_store.cc:276] 0806 04:01:38.636498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.690717 (+ 54219us) service_pool.cc:227] Handling call
0806 04:01:39.780153 (+1089436us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b54000492d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.780287 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1229617024}
W20260806 04:01:39.831406 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 743501us
I20260806 04:01:39.858741 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:39.769392 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:39.853834 1188 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 896) took 1618 ms (client timeout 2066 ms). Trace:
W20260806 04:01:39.202863 4571 consensus_meta.cc:287] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.511s user 0.006s sys 0.003s
I20260806 04:01:39.818269 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.822721 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:39.856267 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 834883us
I20260806 04:01:39.860141 32746 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:39.815697 32478 rpcz_store.cc:276] 0806 04:01:38.722794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.067116 (+344322us) service_pool.cc:227] Handling call
0806 04:01:39.766671 (+699555us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.861454 32427 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:39.821739 32470 raft_consensus.cc:2379] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 8: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:39.771929 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 875) took 1122 ms. Trace:
I20260806 04:01:39.852617 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:39.773533 1192 rpcz_store.cc:269] 0806 04:01:38.209877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.211734 (+ 1857us) service_pool.cc:227] Handling call
0806 04:01:39.454432 (+1242698us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54006603d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.454609 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":934631168}
I20260806 04:01:39.857574 32477 rpcz_store.cc:276] 0806 04:01:38.656131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.127248 (+471117us) service_pool.cc:227] Handling call
0806 04:01:39.816011 (+688763us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.859849 1190 raft_consensus.cc:2396] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:01:39.860839 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 877) took 1210 ms. Trace:
I20260806 04:01:39.860569 32743 raft_consensus.cc:2396] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 04:01:39.859663 1188 rpcz_store.cc:269] 0806 04:01:38.233695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.135466 (+901771us) service_pool.cc:227] Handling call
0806 04:01:39.851970 (+716504us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:39.815671 32474 rpcz_store.cc:269] 0806 04:01:37.537913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:37.664298 (+126385us) service_pool.cc:227] Handling call
0806 04:01:39.463303 (+1799005us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b540026fa54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.463451 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2085962496}
I20260806 04:01:39.860674 32609 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 2.
I20260806 04:01:39.863878 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2315) took 1462 ms. Trace:
I20260806 04:01:39.865047 32743 rpcz_store.cc:276] 0806 04:01:38.401784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.817960 (+1416176us) service_pool.cc:227] Handling call
0806 04:01:39.863814 (+ 45854us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.856748 32473 rpcz_store.cc:276] 0806 04:01:38.699736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.127992 (+428256us) service_pool.cc:227] Handling call
0806 04:01:39.811032 (+683040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.863654 32746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2309) took 1630 ms. Trace:
I20260806 04:01:39.866356 32746 rpcz_store.cc:276] 0806 04:01:38.233332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.818451 (+1585119us) service_pool.cc:227] Handling call
0806 04:01:39.863598 (+ 45147us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:39.867609 1190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 771) took 1647 ms (client timeout 2074 ms). Trace:
I20260806 04:01:39.866967 32428 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:39.868116 1190 rpcz_store.cc:269] 0806 04:01:38.224256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.461980 (+1237724us) service_pool.cc:227] Handling call
0806 04:01:39.867548 (+405568us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.862514 1189 rpcz_store.cc:276] 0806 04:01:38.649399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.462320 (+812921us) service_pool.cc:227] Handling call
0806 04:01:39.771867 (+309547us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.818718 4266 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:39.840310 4327 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.819790 4677 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:39.869454 4481 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:39.834144 4667 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.653643 4647 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:39.870962 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 773) took 1326 ms. Trace:
I20260806 04:01:39.768803 4523 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:39.872685 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:39.455900 4008 raft_consensus.cc:3032] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 04:01:39.852998 1187 rpcz_store.cc:269] 0806 04:01:38.223533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.405099 (+181566us) service_pool.cc:227] Handling call
0806 04:01:39.460612 (+1055513us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b5400611554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.460747 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":452991744}
I20260806 04:01:39.872617 32473 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:39.768381 4414 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:39.878772 1136 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:39.871539 32467 rpcz_store.cc:276] 0806 04:01:38.544392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.870166 (+1325774us) service_pool.cc:227] Handling call
0806 04:01:39.870529 (+ 363us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.870688 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:39.870710 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:39.870748 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:39.870777 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.870823 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:39.870911 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.871850 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.765305 4542 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:39.852648 4659 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:39.863278 1185 rpcz_store.cc:276] 0806 04:01:38.649962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.133998 (+484036us) service_pool.cc:227] Handling call
0806 04:01:39.860785 (+726787us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:39.875752 32470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59514 (request call id 1975) took 1176 ms (client timeout 1554 ms). Trace:
I20260806 04:01:39.767905 4264 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:39.872820 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 774) took 1328 ms. Trace:
I20260806 04:01:39.854553 4638 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:39.876544 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1980) took 1176 ms. Trace:
I20260806 04:01:39.878103 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:39.464102 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 867) took 1241 ms. Trace:
W20260806 04:01:39.837756 4637 consensus_meta.cc:287] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.865s user 0.008s sys 0.000s
I20260806 04:01:39.763335 32475 rpcz_store.cc:276] 0806 04:01:37.019655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:37.020318 (+ 663us) service_pool.cc:227] Handling call
0806 04:01:37.020709 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:37.252456 (+231747us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:37.252484 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:37.252523 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:37.252554 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.051479 (+1798925us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54003688d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:39.051520 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:39.051635 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":97,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7683,"spinlock_wait_cycles":532227840}
I20260806 04:01:39.771095 4662 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:39.879971 1188 raft_consensus.cc:2379] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:39.879956 32738 raft_consensus.cc:1261] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:39.880713 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.881008 32682 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:39.869592 4007 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:39.772564 4644 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:39.773597 4488 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:39.769753 4430 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:39.832780 4139 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:39.886540 1137 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:39.887028 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:39.887459 1136 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:39.821318 4326 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:39.886519 32470 rpcz_store.cc:269] 0806 04:01:38.699163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.904393 (+205230us) service_pool.cc:227] Handling call
0806 04:01:39.875569 (+971176us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b5400202bd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.875700 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1948038016}
I20260806 04:01:39.888506 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.910193 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 895) took 1676 ms. Trace:
I20260806 04:01:39.887796 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.889940 32428 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:39.890404 32478 rpcz_store.cc:276] 0806 04:01:38.544670 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.872001 (+1327331us) service_pool.cc:227] Handling call
0806 04:01:39.872288 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.872527 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:39.872551 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:39.872589 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:39.872620 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.872671 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:39.872768 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:39.936789 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 50332.2us
I20260806 04:01:39.934854 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.838848 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33182 (request call id 548) took 1254 ms. Trace:
I20260806 04:01:39.822465 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 874) took 1173 ms. Trace:
I20260806 04:01:39.891616 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:39.910753 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:39.938069 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 872) took 1289 ms. Trace:
I20260806 04:01:39.892033 32743 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:01:39.864910 4460 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:39.938508 1187 rpcz_store.cc:276] 0806 04:01:38.648527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.876232 (+1227705us) service_pool.cc:227] Handling call
0806 04:01:39.938007 (+ 61775us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:39.918293 32551 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 1.572s (SENT) [suppressed 19 similar messages]
I20260806 04:01:39.871877 4544 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:39.891304 4439 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:39.894121 32473 rpcz_store.cc:276] 0806 04:01:38.700210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.866801 (+1166591us) service_pool.cc:227] Handling call
0806 04:01:39.867169 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.876184 (+ 9015us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.876483 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.942811 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.774109 1184 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:39.894683 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55048 (request call id 779) took 1348 ms. Trace:
I20260806 04:01:39.871186 4623 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.895570 32477 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:39.774222 32740 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:39.895802 32427 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:39.877890 4400 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:39.892313 4266 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:39.979876 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:39.981020 32470 raft_consensus.cc:2354] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8.
I20260806 04:01:39.886154 4542 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:39.876296 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.985601 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 901) took 1750 ms. Trace:
I20260806 04:01:39.983506 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.985932 1190 rpcz_store.cc:276] 0806 04:01:38.235358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.874340 (+1638982us) service_pool.cc:227] Handling call
0806 04:01:39.985545 (+111205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.939357 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1784) took 1696 ms. Trace:
I20260806 04:01:39.986682 32743 rpcz_store.cc:276] 0806 04:01:38.242489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.867986 (+1625497us) service_pool.cc:227] Handling call
0806 04:01:39.884387 (+ 16401us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.939107 (+ 54720us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.939304 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.895074 1182 rpcz_store.cc:276] 0806 04:01:38.222997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.562946 (+339949us) service_pool.cc:227] Handling call
0806 04:01:39.463902 (+900956us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b540078f5d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:39.464020 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":670442880}
I20260806 04:01:39.898911 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.990413 1185 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:39.938812 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 870) took 1715 ms. Trace:
I20260806 04:01:39.990528 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.990944 1192 rpcz_store.cc:276] 0806 04:01:38.223795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.873108 (+1649313us) service_pool.cc:227] Handling call
0806 04:01:39.938753 (+ 65645us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.991559 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 878) took 1341 ms. Trace:
I20260806 04:01:39.939021 32682 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:39.991887 1190 rpcz_store.cc:276] 0806 04:01:38.650218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.990291 (+1340073us) service_pool.cc:227] Handling call
0806 04:01:39.991507 (+ 1216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.939445 1186 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:39.910367 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1783) took 1705 ms. Trace:
I20260806 04:01:39.946790 4105 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:39.935611 1188 rpcz_store.cc:276] 0806 04:01:38.233400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.871602 (+1638202us) service_pool.cc:227] Handling call
0806 04:01:39.910135 (+ 38533us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.943907 32475 raft_consensus.cc:2379] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:39.944269 32469 rpcz_store.cc:276] 0806 04:01:38.546072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.857776 (+1311704us) service_pool.cc:227] Handling call
0806 04:01:39.894628 (+ 36852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.871519 4571 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:39.871358 3790 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.944685 1183 rpcz_store.cc:276] 0806 04:01:38.649102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.944996 (+295894us) service_pool.cc:227] Handling call
0806 04:01:39.822396 (+877400us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.993156 4414 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.999370 3790 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.999867 4571 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:39.999629 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 777) took 1051 ms. Trace:
W20260806 04:01:39.992350 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 53412.7us
I20260806 04:01:39.947870 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.998358 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.000990 4648 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.998566 4585 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.001659 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:40.001901 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:39.994938 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:39.843515 4396 consensus_queue.cc:237] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.998914 4414 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:40.000504 32749 raft_consensus.cc:2379] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:39.879137 4008 raft_consensus.cc:729] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:39.952215 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 783) took 1405 ms. Trace:
I20260806 04:01:39.947253 4439 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:39.967926 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.977841 4625 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.942458 32737 rpcz_store.cc:276] 0806 04:01:38.584678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.115986 (+531308us) service_pool.cc:227] Handling call
0806 04:01:39.838767 (+722781us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.943481 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.012171 32469 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:40.012286 1187 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:39.984287 32741 raft_consensus.cc:2396] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:01:40.004002 32608 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:40.008338 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:40.007232 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:39.999363 32471 raft_consensus.cc:2379] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 9: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:40.014257 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:40.014819 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:40.015246 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55048 (request call id 781) took 1468 ms. Trace:
I20260806 04:01:39.996819 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:40.006824 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 665) took 1067 ms. Trace:
I20260806 04:01:40.016315 1188 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:40.009639 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:40.017058 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 778) took 1058 ms. Trace:
I20260806 04:01:40.017409 1192 raft_consensus.cc:2362] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4. Re-sending same reply.
I20260806 04:01:40.017361 1188 rpcz_store.cc:276] 0806 04:01:38.958247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.996534 (+1038287us) service_pool.cc:227] Handling call
0806 04:01:40.017007 (+ 20473us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:39.999282 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:40.017462 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 765) took 1535 ms. Trace:
I20260806 04:01:39.895076 4667 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:39.919924 4644 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:39.946902 4007 raft_consensus.cc:686] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:39.992985 32738 rpcz_store.cc:276] 0806 04:01:38.205424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.857363 (+1651939us) service_pool.cc:227] Handling call
0806 04:01:39.879733 (+ 22370us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.910127 (+ 30394us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.910316 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.015820 32471 rpcz_store.cc:276] 0806 04:01:38.546665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.986491 (+439826us) service_pool.cc:227] Handling call
0806 04:01:40.015064 (+1028573us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540003e154. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.015191 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2327146240}
W20260806 04:01:39.942391 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 55025.8us
I20260806 04:01:40.016335 32745 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:40.020108 4007 consensus_queue.cc:237] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.018693 32466 rpcz_store.cc:276] 0806 04:01:38.481911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.482019 (+ 108us) service_pool.cc:227] Handling call
0806 04:01:38.958886 (+476867us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:38.959251 (+ 365us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:01:38.959279 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:38.967371 (+ 8092us) log.cc:842] Serialized 26 byte log entry
0806 04:01:39.159082 (+191711us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:39.159123 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.159204 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.005016 (+845812us) raft_consensus.cc:1671] finished
0806 04:01:40.005035 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.017213 (+ 12178us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b54001b1a50. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.017405 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":454,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":5034,"spinlock_wait_cycles":1095886720,"thread_start_us":175907,"threads_started":1,"wal-append.queue_time_us":996569}
I20260806 04:01:39.939543 4675 raft_consensus.cc:2781] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:40.022647 4675 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:40.017970 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 881) took 1076 ms. Trace:
I20260806 04:01:40.023166 32466 raft_consensus.cc:1226] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 04:01:39.942601 4544 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:39.951018 4400 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:39.955075 4679 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:40.023069 4675 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:39.938436 4430 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:39.909494 3697 consensus_queue.cc:1046] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.080s
I20260806 04:01:39.946491 4460 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:39.937525 1137 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:39.994988 4091 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:39.949694 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 775) took 1404 ms. Trace:
W20260806 04:01:39.769318 32322 rpcz_store.cc:269] 0806 04:01:26.951761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:26.966853 (+ 15092us) service_pool.cc:227] Handling call
0806 04:01:39.665326 (+12698473us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:30.852827 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:30.853476 (+ 649us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:30.853527 (+ 51us) write_op.cc:435] Acquired schema lock
0806 04:01:30.853552 (+ 25us) tablet.cc:660] Decoding operations
0806 04:01:30.927388 (+ 73836us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:30.942846 (+ 15458us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:30.965477 (+ 22631us) tablet.cc:683] Acquiring locks for 18 operations
0806 04:01:31.000291 (+ 34814us) tablet.cc:699] Row locks acquired
0806 04:01:31.000322 (+ 31us) write_op.cc:260] PREPARE: finished
0806 04:01:31.000473 (+ 151us) write_op.cc:270] Start()
0806 04:01:31.000660 (+ 187us) write_op.cc:276] Timestamp: P: 1785988891000415 usec, L: 0
0806 04:01:31.000696 (+ 36us) op_driver.cc:331] REPLICATION: starting
0806 04:01:31.001463 (+ 767us) log.cc:842] Serialized 7102 byte log entry
0806 04:01:32.115661 (+1114198us) op_driver.cc:444] REPLICATION: finished
0806 04:01:32.496836 (+381175us) write_op.cc:301] APPLY: starting
0806 04:01:32.496973 (+ 137us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:32.497324 (+ 351us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:32.614226 (+116902us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:33.125112 (+510886us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:33.236217 (+111105us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0806 04:01:33.248039 (+ 11822us) write_op.cc:312] APPLY: finished
0806 04:01:33.306181 (+ 58142us) log.cc:842] Serialized 125 byte log entry
0806 04:01:33.426230 (+120049us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:33.439805 (+ 13575us) write_op.cc:454] Released schema lock
0806 04:01:33.471923 (+ 32118us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:33.524648 (+ 52725us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:33.526469 (+ 1821us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":330715,"apply.run_cpu_time_us":25257,"apply.run_wall_time_us":1049717,"num_ops":18,"prepare.queue_time_us":266385,"prepare.run_cpu_time_us":20199,"prepare.run_wall_time_us":323252,"raft.queue_time_us":252207,"raft.run_cpu_time_us":58351,"raft.run_wall_time_us":313984,"replication_time_us":1106059,"spinlock_wait_cycles":1280,"thread_start_us":799399,"threads_started":5,"wal-append.queue_time_us":349475,"wal-append.run_cpu_time_us":76681,"wal-append.run_wall_time_us":2814299}]]}
I20260806 04:01:39.896729 32321 catalog_manager.cc:5495] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 4 to 7, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 7 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } }
I20260806 04:01:40.191010 32477 rpcz_store.cc:276] 0806 04:01:38.544982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.873426 (+1328444us) service_pool.cc:227] Handling call
0806 04:01:39.893528 (+ 20102us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.949503 (+ 55975us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.949652 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.183588 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1987) took 1260 ms. Trace:
W20260806 04:01:39.949265 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 53543.3us
I20260806 04:01:40.013401 32469 raft_consensus.cc:2932] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:40.188867 32467 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:39.904999 32553 connection.cc:232] RPC call timeout handler was delayed by -2.5343287631403655s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21730000000000002s for a response
I20260806 04:01:40.018986 32474 rpcz_store.cc:276] 0806 04:01:38.547201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.872853 (+1325652us) service_pool.cc:227] Handling call
0806 04:01:39.883825 (+ 10972us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.942186 (+ 58361us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:39.942215 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:39.942250 (+ 35us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:39.942277 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.951969 (+ 9692us) spinlock_profiling.cc:243] Waited 58.1 ms on lock 0x7b540009d654. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:39.952014 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:39.952160 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":133743488}
W20260806 04:01:40.203013 32553 connection.cc:232] RPC call timeout handler was delayed by -2.5001287632621825s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2211s for a response
W20260806 04:01:40.038892 32608 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 2109) took 1493 ms (client timeout 1632 ms). Trace:
I20260806 04:01:40.039467 32610 raft_consensus.cc:1226] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
W20260806 04:01:40.204667 32605 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.027s user 0.022s sys 0.032s
I20260806 04:01:40.044163 32478 raft_consensus.cc:2396] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260806 04:01:40.204384 32608 rpcz_store.cc:269] 0806 04:01:38.545468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.547018 (+ 1550us) service_pool.cc:227] Handling call
0806 04:01:40.038679 (+1491661us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540035cd54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.038815 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3124142464}
I20260806 04:01:40.197902 32466 rpcz_store.cc:276] 0806 04:01:38.923069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.022643 (+1099574us) service_pool.cc:227] Handling call
0806 04:01:40.022941 (+ 298us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.182534 (+159593us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.183229 (+ 695us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:40.206748 32553 connection.cc:232] RPC call timeout handler was delayed by -2.3516287631355226s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2376s for a response
I20260806 04:01:40.018851 32602 rpcz_store.cc:276] 0806 04:01:38.939438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.000883 (+ 61445us) service_pool.cc:227] Handling call
0806 04:01:39.001153 (+ 270us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:39.001471 (+ 318us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:39.013049 (+ 11578us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:39.028748 (+ 15699us) log.cc:842] Serialized 26 byte log entry
0806 04:01:39.185648 (+156900us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:39.185688 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:39.185764 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.006615 (+820851us) raft_consensus.cc:1671] finished
0806 04:01:40.006635 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.006761 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55684,"raft.queue_time_us":14004,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":7131,"thread_start_us":140999,"threads_started":1,"wal-append.queue_time_us":943915}
I20260806 04:01:40.206926 32610 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 658) took 1271 ms. Trace:
I20260806 04:01:39.996057 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:40.207490 32610 rpcz_store.cc:276] 0806 04:01:38.935675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.982089 (+ 46414us) service_pool.cc:227] Handling call
0806 04:01:40.039271 (+1057182us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.205172 (+165901us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.206401 (+ 1229us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540035cd50. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.206654 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2430641920}
W20260806 04:01:40.206923 32605 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.039s user 0.022s sys 0.032s
I20260806 04:01:40.210191 32605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 667) took 1270 ms. Trace:
I20260806 04:01:40.210326 32478 raft_consensus.cc:1226] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:40.204968 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1989) took 1210 ms. Trace:
I20260806 04:01:40.210577 32605 rpcz_store.cc:276] 0806 04:01:38.939952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.028959 (+ 89007us) service_pool.cc:227] Handling call
0806 04:01:39.048474 (+ 19515us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:39.048674 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:39.167937 (+119263us) log.cc:842] Serialized 14 byte log entry
0806 04:01:40.209570 (+1041633us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.209671 (+ 101us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:40.209793 (+ 122us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.209900 (+ 107us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.210135 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":370577,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"thread_start_us":1159775,"threads_started":2,"wal-append.queue_time_us":933901}
I20260806 04:01:40.211064 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1991) took 1150 ms. Trace:
I20260806 04:01:40.013710 32740 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:40.046657 32682 leader_election.cc:398] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:40.210327 32470 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:40.216871 32740 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:40.217244 32470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1993) took 1099 ms. Trace:
I20260806 04:01:40.217924 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1791) took 1951 ms. Trace:
I20260806 04:01:40.017988 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1776) took 1842 ms. Trace:
I20260806 04:01:40.219995 32740 rpcz_store.cc:276] 0806 04:01:38.266677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.215018 (+1948341us) service_pool.cc:227] Handling call
0806 04:01:40.215825 (+ 807us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.217540 (+ 1715us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.217805 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.222857 32740 raft_consensus.cc:1226] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:40.223635 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2321) took 1808 ms. Trace:
I20260806 04:01:40.224030 32740 rpcz_store.cc:276] 0806 04:01:38.415148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.221682 (+1806534us) service_pool.cc:227] Handling call
0806 04:01:40.222294 (+ 612us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.223339 (+ 1045us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.223565 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.224723 32682 leader_election.cc:401] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:01:40.014557 32743 raft_consensus.cc:2396] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 5 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260806 04:01:40.226420 32740 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:40.227085 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2322) took 1811 ms. Trace:
I20260806 04:01:40.018882 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 880) took 1077 ms. Trace:
I20260806 04:01:40.021260 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1777) took 1843 ms. Trace:
I20260806 04:01:40.127265 32746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2310) took 1893 ms. Trace:
I20260806 04:01:40.228421 1182 rpcz_store.cc:276] 0806 04:01:38.941364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.999046 (+1057682us) service_pool.cc:227] Handling call
0806 04:01:40.018822 (+ 19776us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.228901 32746 rpcz_store.cc:276] 0806 04:01:38.233572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.867074 (+1633502us) service_pool.cc:227] Handling call
0806 04:01:39.867349 (+ 275us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:39.868457 (+ 1108us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:39.868487 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:39.883158 (+ 14671us) log.cc:842] Serialized 26 byte log entry
0806 04:01:40.099896 (+216738us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:40.099944 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.100020 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.127010 (+ 26990us) raft_consensus.cc:1671] finished
0806 04:01:40.127035 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.127194 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":794,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":6844,"thread_start_us":149791,"threads_started":1,"wal-append.queue_time_us":213794}
I20260806 04:01:40.230237 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:40.111578 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2306) took 1879 ms. Trace:
I20260806 04:01:40.232481 32748 rpcz_store.cc:276] 0806 04:01:38.232520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.858426 (+1625906us) service_pool.cc:227] Handling call
0806 04:01:39.879021 (+ 20595us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:39.879411 (+ 390us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:39.879439 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:39.887567 (+ 8128us) log.cc:842] Serialized 26 byte log entry
0806 04:01:40.100245 (+212678us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:40.100277 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.100339 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.111295 (+ 10956us) raft_consensus.cc:1671] finished
0806 04:01:40.111316 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.111509 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":83,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":6393,"thread_start_us":120304,"threads_started":1,"wal-append.queue_time_us":198749}
I20260806 04:01:40.016005 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:40.234021 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:40.049036 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 111595us
I20260806 04:01:40.220489 32741 rpcz_store.cc:276] 0806 04:01:38.175261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.773714 (+1598453us) service_pool.cc:227] Handling call
0806 04:01:40.017787 (+244073us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540011cbd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.017922 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300597120}
I20260806 04:01:39.998979 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:40.041168 32738 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:40.237713 1137 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:39.950214 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:40.257628 4656 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.143639 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 855) took 1932 ms. Trace:
I20260806 04:01:40.257068 4625 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:40.023689 1192 rpcz_store.cc:276] 0806 04:01:38.941674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.009364 (+1067690us) service_pool.cc:227] Handling call
0806 04:01:40.017901 (+ 8537us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.260180 4625 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.261291 1190 rpcz_store.cc:276] 0806 04:01:38.211546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.019501 (+1807955us) service_pool.cc:227] Handling call
0806 04:01:40.019805 (+ 304us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.020160 (+ 355us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.020187 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:40.020590 (+ 403us) log.cc:842] Serialized 26 byte log entry
0806 04:01:40.143274 (+122684us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:40.143319 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.143384 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.143424 (+ 40us) raft_consensus.cc:1671] finished
0806 04:01:40.143434 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.143571 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":912,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":122397,"threads_started":1,"wal-append.queue_time_us":119602}
I20260806 04:01:40.262418 1137 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.263324 4625 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:39.999993 3790 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:40.263960 4266 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:40.047329 4091 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:40.264578 4266 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:39.951656 4326 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:40.261453 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2327) took 1197 ms. Trace:
I20260806 04:01:40.265555 4326 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:40.266237 4326 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.267275 4396 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:40.043045 4400 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.228391 32745 rpcz_store.cc:276] 0806 04:01:38.178232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.856746 (+1678514us) service_pool.cc:227] Handling call
0806 04:01:40.016164 (+159418us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.020587 (+ 4423us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.021097 (+ 510us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54000bd450. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.021206 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":307842816}
I20260806 04:01:40.270433 32745 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260806 04:01:40.270707 32745 pending_rounds.cc:76] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Aborting all ops after (but not including) 3
I20260806 04:01:40.270941 32745 pending_rounds.cc:97] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20260806 04:01:40.271399 32745 raft_consensus.cc:2894] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20260806 04:01:40.059414 4414 raft_consensus.cc:3032] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:40.012018 4439 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.060883 4460 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.259619 4182 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:39.951368 4481 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:39.956907 1184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 859) took 1736 ms (client timeout 2059 ms). Trace:
I20260806 04:01:40.000607 1185 rpcz_store.cc:276] 0806 04:01:38.947741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.886292 (+938551us) service_pool.cc:227] Handling call
0806 04:01:39.999570 (+113278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.008034 4523 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.053768 1186 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:40.264377 3790 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:40.025214 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 770) took 1806 ms. Trace:
I20260806 04:01:40.272475 4414 raft_consensus.cc:729] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:40.278242 1187 rpcz_store.cc:276] 0806 04:01:38.219121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.946260 (+1727139us) service_pool.cc:227] Handling call
0806 04:01:40.025155 (+ 78895us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.094839 4538 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.264976 4091 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:40.284149 4538 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.018326 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:40.285264 4523 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.285532 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 882) took 1339 ms. Trace:
I20260806 04:01:40.286008 1188 rpcz_store.cc:276] 0806 04:01:38.945890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.018106 (+1072216us) service_pool.cc:227] Handling call
0806 04:01:40.285299 (+267193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.287098 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:40.260987 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:40.287732 1188 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.040810 4544 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.013603 4008 consensus_queue.cc:260] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.288903 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:40.289135 4008 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:40.289628 1192 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:39.995473 4105 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:40.273104 4182 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:39.937824 4139 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:40.291390 4182 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:40.291896 4139 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:40.273772 1184 rpcz_store.cc:269] 0806 04:01:38.220806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.476061 (+255255us) service_pool.cc:227] Handling call
0806 04:01:39.049113 (+573052us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4bce 000000000034aea9 00007fd172120a30 00007fd1721e67b4 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:39.953052 (+903939us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57271680}
I20260806 04:01:40.295083 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:39.886472 4581 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260806 04:01:40.040014 4644 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.274801 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:40.297015 1185 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:39.947256 4632 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:40.047919 4647 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:40.299667 4647 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:40.300221 4647 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:39.898475 4637 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.291333 4105 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.306519 4637 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:39.897617 4659 raft_consensus.cc:686] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:40.122625 4675 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.308749 4659 consensus_queue.cc:237] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.309732 4675 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.131758 4488 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.123653 4430 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.133136 4560 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.312470 4488 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.313596 32612 raft_consensus.cc:2379] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:40.279544 4414 consensus_queue.cc:260] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.314205 4560 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.314846 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42618 (request call id 660) took 1378 ms. Trace:
I20260806 04:01:40.139766 4542 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.236186 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:40.211562 32474 rpcz_store.cc:276] 0806 04:01:38.994799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.203265 (+1208466us) service_pool.cc:227] Handling call
0806 04:01:40.203878 (+ 613us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.204403 (+ 525us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.204462 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.204542 (+ 80us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:40.204573 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.204627 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.204749 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.295876 1184 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
I20260806 04:01:40.315879 1182 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:40.315478 32739 raft_consensus.cc:1226] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:40.316359 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.212596 32478 rpcz_store.cc:276] 0806 04:01:39.060719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.209345 (+1148626us) service_pool.cc:227] Handling call
0806 04:01:40.209893 (+ 548us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.210843 (+ 950us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.211016 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:40.225800 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 179261us
I20260806 04:01:40.317247 32474 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.317706 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2307) took 2084 ms. Trace:
I20260806 04:01:40.318137 32739 rpcz_store.cc:276] 0806 04:01:38.232773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.858639 (+1625866us) service_pool.cc:227] Handling call
0806 04:01:39.865266 (+ 6627us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.316705 (+451439us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.317407 (+ 702us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b5400244354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:40.317631 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1034160384}
I20260806 04:01:40.208066 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1990) took 1176 ms. Trace:
I20260806 04:01:40.320190 32739 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260806 04:01:40.320542 4542 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.320788 32739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2331) took 1255 ms. Trace:
I20260806 04:01:40.316154 4430 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:40.321048 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.239315 32740 rpcz_store.cc:276] 0806 04:01:38.415443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.225485 (+1810042us) service_pool.cc:227] Handling call
0806 04:01:40.226019 (+ 534us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.226861 (+ 842us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.227042 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.238798 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2326) took 1174 ms. Trace:
I20260806 04:01:40.321764 32741 rpcz_store.cc:276] 0806 04:01:39.064185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.237561 (+1173376us) service_pool.cc:227] Handling call
0806 04:01:40.238035 (+ 474us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.238476 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.238500 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.238566 (+ 66us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:40.238602 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.238655 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.238749 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.315338 32612 rpcz_store.cc:276] 0806 04:01:38.936419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:38.975588 (+ 39169us) service_pool.cc:227] Handling call
0806 04:01:40.314541 (+1338953us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540007eed4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.314720 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2935844608}
I20260806 04:01:40.217679 32470 rpcz_store.cc:276] 0806 04:01:39.118079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.209462 (+1091383us) service_pool.cc:227] Handling call
0806 04:01:40.209935 (+ 473us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.216894 (+ 6959us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.217163 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11136}
I20260806 04:01:40.238694 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1785) took 1978 ms. Trace:
I20260806 04:01:40.236297 32749 raft_consensus.cc:1226] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:40.231974 32746 raft_consensus.cc:1226] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260806 04:01:40.246522 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 328086us
W20260806 04:01:40.251224 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 361123us
I20260806 04:01:40.228349 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2324) took 1740 ms. Trace:
I20260806 04:01:40.194177 32321 catalog_manager.cc:5495] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:40.255965 4679 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:40.257078 4571 consensus_queue.cc:1046] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:40.259323 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 779) took 1275 ms. Trace:
I20260806 04:01:40.267072 4266 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:40.267992 4396 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.351540 4396 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.272059 32745 raft_consensus.cc:2866] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:40.353245 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2330) took 1288 ms. Trace:
I20260806 04:01:40.353602 32745 rpcz_store.cc:276] 0806 04:01:39.065171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.269362 (+1204191us) service_pool.cc:227] Handling call
0806 04:01:40.269844 (+ 482us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.352987 (+ 83143us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.353186 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:40.268285 32553 connection.cc:232] RPC call timeout handler was delayed by -1.4957287632860243s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3327s for a response
W20260806 04:01:40.356741 32553 connection.cc:232] RPC call timeout handler was delayed by -1.2535552571061999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
W20260806 04:01:40.357110 32553 connection.cc:232] RPC call timeout handler was delayed by -1.2409552570898086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260806 04:01:40.277611 3790 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.265461 32748 rpcz_store.cc:276] 0806 04:01:39.064457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.260165 (+1195708us) service_pool.cc:227] Handling call
0806 04:01:40.260692 (+ 527us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.261075 (+ 383us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.261107 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.261167 (+ 60us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:40.261202 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.261263 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.261393 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:01:40.277731 1186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 871) took 2053 ms (client timeout 2247 ms). Trace:
I20260806 04:01:40.315307 4414 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.321486 32739 rpcz_store.cc:276] 0806 04:01:39.065391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.319331 (+1253940us) service_pool.cc:227] Handling call
0806 04:01:40.319707 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.320571 (+ 864us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.320743 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.272471 1137 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.326308 32466 rpcz_store.cc:276] 0806 04:01:39.031550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.206827 (+1175277us) service_pool.cc:227] Handling call
0806 04:01:40.207302 (+ 475us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.207715 (+ 413us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.207742 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.207809 (+ 67us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:40.207858 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.207915 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.208008 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.326412 32738 rpcz_store.cc:276] 0806 04:01:38.264961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.019759 (+1754798us) service_pool.cc:227] Handling call
0806 04:01:40.020057 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.238171 (+218114us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.238495 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.317234 1184 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:40.338668 32749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 553) took 1620 ms. Trace:
I20260806 04:01:40.319945 32682 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.341329 32746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2325) took 1769 ms. Trace:
I20260806 04:01:40.362746 32746 rpcz_store.cc:276] 0806 04:01:38.571760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.230993 (+1659233us) service_pool.cc:227] Handling call
0806 04:01:40.231551 (+ 558us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.341044 (+109493us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.341267 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.339506 32743 rpcz_store.cc:276] 0806 04:01:38.487358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.226982 (+1739624us) service_pool.cc:227] Handling call
0806 04:01:40.227464 (+ 482us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.227871 (+ 407us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.227904 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.227964 (+ 60us) raft_consensus.cc:1631] Marking committed up to 1
0806 04:01:40.227990 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.228037 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.228166 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.363413 32474 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:40.346282 32470 raft_consensus.cc:1226] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:40.343202 1189 rpcz_store.cc:276] 0806 04:01:38.983720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.946062 (+962342us) service_pool.cc:227] Handling call
0806 04:01:40.258632 (+312570us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.347361 32321 catalog_manager.cc:5495] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
W20260806 04:01:40.357470 32553 connection.cc:232] RPC call timeout handler was delayed by -1.2310552571434528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
I20260806 04:01:40.363654 32749 rpcz_store.cc:276] 0806 04:01:38.713591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.235337 (+1521746us) service_pool.cc:227] Handling call
0806 04:01:40.235840 (+ 503us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.333519 (+ 97679us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.333735 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.367349 4571 raft_consensus.cc:3032] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Stepping down as leader of term 7
W20260806 04:01:40.370584 32553 connection.cc:232] RPC call timeout handler was delayed by -1.2310552571434528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
W20260806 04:01:40.360028 1186 rpcz_store.cc:269] 0806 04:01:38.224042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.871707 (+1647665us) service_pool.cc:227] Handling call
0806 04:01:40.277521 (+405814us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.348210 4679 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:40.375931 32553 connection.cc:232] RPC call timeout handler was delayed by -1.1977552571333945s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20260806 04:01:40.367839 4571 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:40.381812 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:40.383277 32466 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: []
W20260806 04:01:40.388227 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 116009us
I20260806 04:01:40.388813 4542 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:40.394655 32467 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:40.320585 4671 consensus_queue.cc:1046] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260806 04:01:40.364431 4396 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:40.400502 4625 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:40.400795 32470 raft_consensus.cc:1226] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 04:01:40.401417 32466 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260806 04:01:40.404917 32321 catalog_manager.cc:5495] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260806 04:01:40.401674 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 81941.3us
I20260806 04:01:40.379499 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 861) took 2154 ms. Trace:
I20260806 04:01:40.402302 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:40.404218 32428 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.406418 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:40.409068 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:40.426625 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 862) took 2204 ms. Trace:
I20260806 04:01:40.429284 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1986) took 1504 ms. Trace:
I20260806 04:01:40.429654 1186 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.428287 1188 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:40.421705 1184 rpcz_store.cc:276] 0806 04:01:38.221393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.316254 (+2094861us) service_pool.cc:227] Handling call
0806 04:01:40.316742 (+ 488us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.362243 (+ 45501us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.373377 (+ 11134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.434829 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:40.437072 32553 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.437306 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 864) took 2215 ms. Trace:
I20260806 04:01:40.437712 1184 rpcz_store.cc:276] 0806 04:01:38.222194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.436116 (+2213922us) service_pool.cc:227] Handling call
0806 04:01:40.436570 (+ 454us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.436850 (+ 280us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.436886 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.436954 (+ 68us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:40.437027 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.437104 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.437228 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.430146 32471 rpcz_store.cc:276] 0806 04:01:38.922821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.019989 (+1097168us) service_pool.cc:227] Handling call
0806 04:01:40.020324 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.020621 (+ 297us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.020647 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:40.021040 (+ 393us) log.cc:842] Serialized 26 byte log entry
0806 04:01:40.376915 (+355875us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:40.386715 (+ 9800us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.392259 (+ 5544us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.418525 (+ 26266us) raft_consensus.cc:1671] finished
0806 04:01:40.418540 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.427528 (+ 8988us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4950,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":5534,"thread_start_us":328128,"threads_started":1,"wal-append.queue_time_us":262113}
I20260806 04:01:40.449222 32427 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:40.450690 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 64663.9us
I20260806 04:01:40.452296 4396 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.453187 32321 catalog_manager.cc:5495] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:40.454735 4396 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.439957 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 772) took 2215 ms. Trace:
I20260806 04:01:40.461453 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.456022 32321 catalog_manager.cc:5495] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:40.462991 4400 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.464257 1189 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.464705 32475 raft_consensus.cc:2379] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:40.463418 32321 catalog_manager.cc:5495] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129), VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130), VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131), VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 4 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:40.466750 32477 raft_consensus.cc:1226] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 04:01:40.443765 4637 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:40.468551 4647 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.469278 4105 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.469771 1190 raft_consensus.cc:1226] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:01:40.471074 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 856) took 2256 ms. Trace:
I20260806 04:01:40.460284 4264 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:40.472368 4264 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:40.472745 4264 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
W20260806 04:01:40.476677 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 72770.5us
I20260806 04:01:40.477727 32428 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:40.479017 32428 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.479154 4647 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.468051 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1988) took 1531 ms. Trace:
I20260806 04:01:40.451840 32427 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:40.466187 32321 catalog_manager.cc:5495] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:40.471388 1190 rpcz_store.cc:276] 0806 04:01:38.214530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.263599 (+2049069us) service_pool.cc:227] Handling call
0806 04:01:40.273459 (+ 9860us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.470277 (+196818us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.470828 (+ 551us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54006405d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:40.470999 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":450912000}
I20260806 04:01:40.480401 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:40.480372 32477 rpcz_store.cc:276] 0806 04:01:38.936358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.193137 (+1256779us) service_pool.cc:227] Handling call
0806 04:01:40.466528 (+273391us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.467208 (+ 680us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.467851 (+ 643us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540009cc50. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.467993 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":626994432}
W20260806 04:01:40.487950 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 87945.6us
I20260806 04:01:40.491834 1185 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:40.490872 1192 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
I20260806 04:01:40.461018 4091 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.495175 1188 raft_consensus.cc:1226] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:40.495328 4460 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:40.471617 4105 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.495623 4581 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 04:01:40.496031 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 59149.2us
W20260806 04:01:40.493402 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 49662.5us
I20260806 04:01:40.495975 4560 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:40.507665 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 773) took 2282 ms. Trace:
I20260806 04:01:40.495728 32466 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:40.508253 1188 rpcz_store.cc:276] 0806 04:01:38.224825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.442952 (+2218127us) service_pool.cc:227] Handling call
0806 04:01:40.491464 (+ 48512us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.507321 (+ 15857us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.507603 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.493162 4675 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:40.511128 4581 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:40.537328 4581 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.524318 4671 consensus_queue.cc:1046] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260806 04:01:40.505842 4460 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:40.511786 4091 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:40.508720 4560 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:40.512579 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:40.512691 4439 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.513024 4266 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.505618 4571 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.504594 4486 raft_consensus.cc:3032] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:40.514765 4481 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:40.515384 4656 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:40.552203 4486 raft_consensus.cc:729] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:40.517522 4182 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.553340 4486 consensus_queue.cc:260] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.553952 32551 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:40.554505 4486 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:40.554831 4182 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.504386 4008 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:40.523440 4430 consensus_queue.cc:1046] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:40.556367 1185 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:40.556705 4430 raft_consensus.cc:3032] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:40.557222 4430 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:40.520872 1190 raft_consensus.cc:1226] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20260806 04:01:40.456532 3790 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.558255 4430 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.512609 32737 raft_consensus.cc:2354] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8.
I20260806 04:01:40.559726 3790 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.489518 4139 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.525107 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.561859 32682 leader_election.cc:398] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:40.527741 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:40.562592 32602 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:40.556741 32610 raft_consensus.cc:1226] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:40.563645 4656 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:01:40.527895 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:40.524211 4326 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.528769 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:40.566587 4571 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:40.529278 4544 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.444748 4625 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.567241 32551 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:40.569696 4625 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.570489 4008 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:40.571123 4008 raft_consensus.cc:686] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:40.572203 4008 consensus_queue.cc:237] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.572621 4538 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:40.503479 4637 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:40.520069 1186 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:40.533852 1189 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:40.525622 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:40.576382 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 886) took 2345 ms. Trace:
I20260806 04:01:40.577075 32605 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:40.577620 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 52123.3us
I20260806 04:01:40.578248 4326 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.578534 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.579039 4637 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:40.579474 4637 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:40.575415 4105 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:40.580260 4105 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:40.580427 4007 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:40.580632 4105 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:40.542078 1192 raft_consensus.cc:1226] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 04:01:40.542585 4460 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.545744 1137 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.542765 4560 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:40.547639 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:40.547955 4439 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:40.548406 4266 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.548707 1183 raft_consensus.cc:2379] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:40.549436 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:40.549690 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:40.550274 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:40.551319 32603 pending_rounds.cc:76] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 7
I20260806 04:01:40.554929 4488 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:40.544003 4671 raft_consensus.cc:3032] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:40.577632 4523 raft_consensus.cc:2781] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:01:40.460558 1184 rpcz_store.cc:276] 0806 04:01:38.224556 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.438805 (+2214249us) service_pool.cc:227] Handling call
0806 04:01:40.439238 (+ 433us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.439608 (+ 370us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.439634 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.439682 (+ 48us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:40.439715 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.439774 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.439901 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.592409 4488 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:40.593187 4488 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:40.593647 1184 raft_consensus.cc:1226] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:40.594303 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 897) took 2360 ms. Trace:
I20260806 04:01:40.594648 1184 rpcz_store.cc:276] 0806 04:01:38.233921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.593013 (+2359092us) service_pool.cc:227] Handling call
0806 04:01:40.593427 (+ 414us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.594053 (+ 626us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.594248 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.561769 4139 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.430037 1182 rpcz_store.cc:276] 0806 04:01:38.221670 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.316700 (+2095030us) service_pool.cc:227] Handling call
0806 04:01:40.317163 (+ 463us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.317506 (+ 343us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.317542 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:40.318139 (+ 597us) log.cc:842] Serialized 26 byte log entry
0806 04:01:40.387103 (+ 68964us) raft_consensus.cc:1631] Marking committed up to 10
0806 04:01:40.391686 (+ 4583us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.409746 (+ 18060us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.426310 (+ 16564us) raft_consensus.cc:1671] finished
0806 04:01:40.426333 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.426525 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":60467,"threads_started":1,"wal-append.queue_time_us":106025}
I20260806 04:01:40.525475 4400 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.599387 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 793) took 1876 ms. Trace:
I20260806 04:01:40.599205 4266 consensus_queue.cc:1046] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:40.600178 1184 raft_consensus.cc:1226] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: [5.6-6.7]
I20260806 04:01:40.600823 4481 raft_consensus.cc:3032] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:40.592483 4671 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:40.601897 4481 raft_consensus.cc:729] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:40.557703 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 888) took 2326 ms. Trace:
W20260806 04:01:40.575812 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 96903.6us
I20260806 04:01:40.582489 1188 raft_consensus.cc:1226] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:40.565678 32683 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:40.576778 1190 rpcz_store.cc:276] 0806 04:01:38.230892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.489776 (+2258884us) service_pool.cc:227] Handling call
0806 04:01:40.520543 (+ 30767us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:40.576115 (+ 55572us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.576318 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.585485 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 889) took 2353 ms. Trace:
I20260806 04:01:40.587149 32603 pending_rounds.cc:97] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260806 04:01:40.587878 32609 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:40.584151 4544 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.590344 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:40.592832 32682 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:40.599773 32473 rpcz_store.cc:276] 0806 04:01:38.723069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.012494 (+1289425us) service_pool.cc:227] Handling call
0806 04:01:40.012795 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.013364 (+ 569us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.013394 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:40.013818 (+ 424us) log.cc:842] Serialized 26 byte log entry
0806 04:01:40.410117 (+396299us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:40.431106 (+ 20989us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.445210 (+ 14104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.598878 (+153668us) raft_consensus.cc:1671] finished
0806 04:01:40.598999 (+ 121us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.599207 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":109,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":351750,"threads_started":1,"wal-append.queue_time_us":414802}
W20260806 04:01:40.586143 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 93566.5us
I20260806 04:01:40.564800 4430 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.563019 4264 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.605546 4671 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.603574 1185 rpcz_store.cc:276] 0806 04:01:38.231421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.505802 (+2274381us) service_pool.cc:227] Handling call
0806 04:01:40.524800 (+ 18998us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.556785 (+ 31985us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.557444 (+ 659us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b540013ddd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:40.557638 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71908096}
I20260806 04:01:40.616312 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 891) took 2384 ms. Trace:
I20260806 04:01:40.616641 1188 rpcz_store.cc:276] 0806 04:01:38.232340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.509518 (+2277178us) service_pool.cc:227] Handling call
0806 04:01:40.581479 (+ 71961us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.605749 (+ 24270us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.616112 (+ 10363us) spinlock_profiling.cc:243] Waited 71.6 ms on lock 0x7b5400480650. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.616258 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164642688}
I20260806 04:01:40.618093 1188 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260806 04:01:40.616248 4671 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.601188 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 776) took 1653 ms. Trace:
I20260806 04:01:40.598583 32605 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:40.606767 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 905) took 1595 ms. Trace:
I20260806 04:01:40.592670 4523 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:40.603650 32603 raft_consensus.cc:2866] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:40.608300 4139 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:40.601393 4460 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.608505 32682 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:40.608781 4481 consensus_queue.cc:260] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.607925 32608 raft_consensus.cc:2379] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:40.585089 4182 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:40.610818 32553 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.611418 4538 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:01:40.612712 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 47275us
I20260806 04:01:40.604501 1192 rpcz_store.cc:276] 0806 04:01:38.231691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.506181 (+2274490us) service_pool.cc:227] Handling call
0806 04:01:40.540031 (+ 33850us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.584665 (+ 44634us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.585430 (+ 765us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1414528}
I20260806 04:01:40.614147 4091 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:40.615135 32428 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.618693 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 884) took 1558 ms. Trace:
I20260806 04:01:40.647608 1188 rpcz_store.cc:276] 0806 04:01:39.060605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.617500 (+1556895us) service_pool.cc:227] Handling call
0806 04:01:40.617876 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.618471 (+ 595us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.618645 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.627947 32609 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:40.628983 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.627569 1183 rpcz_store.cc:276] 0806 04:01:39.011812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:39.998735 (+986923us) service_pool.cc:227] Handling call
0806 04:01:40.606513 (+607778us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b5400611554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:40.606689 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":714433792}
I20260806 04:01:40.648973 4659 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:40.649371 4659 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:40.649516 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.634816 1189 raft_consensus.cc:1226] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:40.649785 4659 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:40.650866 1137 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.651028 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 894) took 2417 ms. Trace:
I20260806 04:01:40.651423 1189 rpcz_store.cc:276] 0806 04:01:38.233130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.586037 (+2352907us) service_pool.cc:227] Handling call
0806 04:01:40.588852 (+ 2815us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.650190 (+ 61338us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.650807 (+ 617us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b54004f10d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:40.650971 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94726912}
I20260806 04:01:40.653082 1188 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 8 LEARNER]: Advancing to term 9
I20260806 04:01:40.629920 4523 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:40.656514 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.630102 4139 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:40.657874 32551 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:40.635314 4538 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:40.649884 1183 raft_consensus.cc:1226] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:40.613464 4581 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.669826 4544 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:40.675594 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 64837.5us
I20260806 04:01:40.661897 4647 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:40.626857 1184 rpcz_store.cc:276] 0806 04:01:38.947513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.595523 (+1648010us) service_pool.cc:227] Handling call
0806 04:01:40.595954 (+ 431us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:40.600877 (+ 4923us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.601115 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.629972 4679 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.666409 4139 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.663444 4625 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:40.685014 1192 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260806 04:01:40.673291 4538 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:40.690202 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 885) took 1623 ms. Trace:
I20260806 04:01:40.692373 1192 rpcz_store.cc:276] 0806 04:01:39.066851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.637123 (+1570272us) service_pool.cc:227] Handling call
0806 04:01:40.684786 (+ 47663us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.688903 (+ 4117us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.690142 (+ 1239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:40.671563 4656 raft_consensus.cc:2726] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:40.675601 4007 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:01:40.695371 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 80402.9us
I20260806 04:01:40.692250 1137 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.674897 4091 raft_consensus.cc:686] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:40.678340 4400 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:40.724220 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 115844us
I20260806 04:01:40.725152 32682 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:40.724869 4091 consensus_queue.cc:237] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.698390 4264 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.729169 4486 consensus_queue.cc:1046] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 04:01:40.730427 4656 raft_consensus.cc:3032] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:40.730850 4656 raft_consensus.cc:729] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:40.699322 4625 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.731871 4656 consensus_queue.cc:260] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.733039 4656 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.733642 4625 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.701720 4560 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.736023 4560 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.715595 4659 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.703181 4007 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:40.736794 4007 raft_consensus.cc:686] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:40.737727 4007 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.665313 4396 consensus_queue.cc:1046] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260806 04:01:40.686303 4542 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:01:40.703416 4647 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:40.740814 4396 raft_consensus.cc:3032] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:40.740725 4647 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.741227 4396 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:40.742134 4396 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.704355 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:40.743471 4396 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:40.742743 4647 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:40.707171 32683 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:40.747340 4625 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:40.708673 4581 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.695871 4481 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:40.673763 4637 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:40.720347 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 91204.1us
I20260806 04:01:40.720913 4460 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:40.721912 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 46384.5us
I20260806 04:01:40.722481 4266 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:40.722776 4644 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.704442 32428 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:40.747404 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:40.758904 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 101118us
W20260806 04:01:40.751560 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 58984.8us
I20260806 04:01:40.723778 4679 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.762740 32551 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:40.763702 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:40.749130 4625 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:40.768648 32478 raft_consensus.cc:2379] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:40.769346 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 44366.4us
I20260806 04:01:40.769024 32427 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:40.774449 1188 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260806 04:01:40.763070 4488 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:40.775411 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 71071.1us
I20260806 04:01:40.775956 32474 raft_consensus.cc:2379] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 8: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:40.776350 32428 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.777122 4488 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.777791 32612 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:40.778312 1137 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.778556 4008 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:40.779172 4008 raft_consensus.cc:686] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:40.780189 4008 consensus_queue.cc:237] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.781270 4638 raft_consensus.cc:2726] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:40.782001 4638 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:40.786044 4139 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.788901 4647 raft_consensus.cc:2932] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:40.790642 4523 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.779963 4460 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:40.790014 4644 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.797226 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.798633 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:40.799834 32427 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:40.759106 4266 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:40.800596 4544 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:40.801026 4544 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:40.801378 4544 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:40.801685 4637 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.801908 4326 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:40.803220 4326 raft_consensus.cc:686] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:40.800622 4266 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:40.804219 4326 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:40.806183 4734 log.cc:924] Time spent T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
I20260806 04:01:40.808538 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:40.774731 1188 pending_rounds.cc:76] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Aborting all ops after (but not including) 7
I20260806 04:01:40.810938 1188 pending_rounds.cc:97] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260806 04:01:40.765372 4486 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:40.796417 4581 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:40.760283 4675 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:40.801493 4679 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:40.771386 4625 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:40.821619 1188 raft_consensus.cc:2866] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:40.827908 32553 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:40.828440 4560 consensus_queue.cc:1046] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20260806 04:01:40.809461 4460 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:40.829424 4460 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.830420 32682 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.810271 4139 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.834525 4671 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:40.834740 4560 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:40.820905 4486 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:40.836519 4581 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260806 04:01:40.839460 4560 raft_consensus.cc:686] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:40.830212 4400 consensus_queue.cc:1046] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260806 04:01:40.837898 4671 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:40.835408 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 906) took 1817 ms. Trace:
I20260806 04:01:40.839242 4637 raft_consensus.cc:3032] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:40.830806 4679 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:40.855923 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 47494.8us
I20260806 04:01:40.858258 4637 raft_consensus.cc:729] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:40.853511 1185 rpcz_store.cc:276] 0806 04:01:39.018171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.616726 (+1598555us) service_pool.cc:227] Handling call
0806 04:01:40.617145 (+ 419us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.617453 (+ 308us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:40.628573 (+ 11120us) log.cc:842] Serialized 14 byte log entry
0806 04:01:40.834944 (+206371us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.835005 (+ 61us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:40.835080 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.835144 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.835347 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":67020,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":2006,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"thread_start_us":171782,"threads_started":2}
I20260806 04:01:40.833702 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 883) took 1878 ms. Trace:
W20260806 04:01:40.858515 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 50085.5us
I20260806 04:01:40.851436 4486 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.839020 4644 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.020s
I20260806 04:01:40.857553 4671 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.858543 4679 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.872454 32682 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:40.873871 4746 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 04:01:40.874563 4742 log.cc:924] Time spent T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260806 04:01:40.876106 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:40.878185 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:40.879722 32682 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:40.880636 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:40.881330 4266 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.882437 4139 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:40.882836 4139 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:40.883198 4139 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:40.883594 4266 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:40.883764 32740 raft_consensus.cc:1226] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 2. Current term is 3. Ops: []
I20260806 04:01:40.884112 4659 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:40.884569 4720 log.cc:924] Time spent T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.231s user 0.001s sys 0.000s
I20260806 04:01:40.885852 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 858) took 2665 ms. Trace:
I20260806 04:01:40.891469 32427 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:40.885893 4430 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:40.886351 1187 rpcz_store.cc:276] 0806 04:01:38.220534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.279094 (+2058560us) service_pool.cc:227] Handling call
0806 04:01:40.279603 (+ 509us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.280205 (+ 602us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:40.280230 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:40.313507 (+ 33277us) log.cc:842] Serialized 26 byte log entry
0806 04:01:40.379724 (+ 66217us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:40.379765 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.379836 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:40.885639 (+505803us) raft_consensus.cc:1671] finished
0806 04:01:40.885657 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.885773 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12217,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":176,"thread_start_us":52500,"threads_started":1,"wal-append.queue_time_us":129735}
I20260806 04:01:40.864450 4105 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.898407 4523 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.899370 1186 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:01:40.868357 4560 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.901087 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 892) took 2668 ms. Trace:
I20260806 04:01:40.901403 1186 rpcz_store.cc:276] 0806 04:01:38.232634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.585265 (+2352631us) service_pool.cc:227] Handling call
0806 04:01:40.589383 (+ 4118us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.899788 (+310405us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.900826 (+ 1038us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54006103d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:40.901011 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":712363904}
I20260806 04:01:40.903635 4105 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:40.904359 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 1985) took 1943 ms. Trace:
I20260806 04:01:40.915334 32749 raft_consensus.cc:1226] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:40.915633 4139 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:40.916723 4750 log.cc:924] Time spent T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.054s user 0.003s sys 0.001s
I20260806 04:01:40.917356 4414 raft_consensus.cc:3032] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:40.917770 4414 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:40.923766 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 54913.3us
I20260806 04:01:40.908274 32469 rpcz_store.cc:276] 0806 04:01:38.922547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.012810 (+1090263us) service_pool.cc:227] Handling call
0806 04:01:40.013115 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.013296 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:01:40.339442 (+326146us) log.cc:842] Serialized 14 byte log entry
0806 04:01:40.849307 (+509865us) log.cc:842] Serialized 14 byte log entry
0806 04:01:40.850437 (+ 1130us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.854102 (+ 3665us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:40.860888 (+ 6786us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.860957 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.865758 (+ 4801us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":492,"prepare.queue_time_us":403847,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":13721,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":1491,"thread_start_us":352283,"threads_started":2,"wal-append.queue_time_us":115541}
I20260806 04:01:40.933043 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.933554 32473 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
W20260806 04:01:40.936725 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 75090us
I20260806 04:01:40.949935 32466 raft_consensus.cc:1226] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:40.953600 32553 leader_election.cc:302] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:40.873994 4745 consensus_queue.cc:1046] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260806 04:01:40.928609 4414 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.967984 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:40.925957 4523 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260806 04:01:40.966499 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:40.955577 4139 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:40.971530 32749 raft_consensus.cc:2379] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 7: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:40.969367 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:40.974781 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.975870 4760 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election won for term 9
W20260806 04:01:40.977010 4747 log.cc:924] Time spent T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260806 04:01:40.979466 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:40.980846 32743 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 04:01:40.956007 4637 consensus_queue.cc:260] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:40.985090 4637 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:40.985759 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:40.986455 32603 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:40.988595 4007 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:40.988979 4007 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:40.990069 4007 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.990837 32605 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:40.991389 4007 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:40.993243 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:40.993474 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:40.994117 32740 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:40.995502 4760 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:40.996564 4760 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:40.997608 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:40.998245 32614 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:40.993347 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:40.999703 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:40.861193 1182 rpcz_store.cc:276] 0806 04:01:38.954514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.597334 (+1642820us) service_pool.cc:227] Handling call
0806 04:01:40.597761 (+ 427us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:40.598069 (+ 308us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:01:40.658130 (+ 60061us) log.cc:842] Serialized 14 byte log entry
0806 04:01:40.772308 (+114178us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:40.793937 (+ 21629us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:40.820996 (+ 27059us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.825466 (+ 4470us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:40.832919 (+ 7453us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":867,"prepare.queue_time_us":55554,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":18765,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"thread_start_us":101623,"threads_started":2,"wal-append.queue_time_us":151843}
I20260806 04:01:41.000367 32608 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 4
I20260806 04:01:41.003016 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:41.003809 32473 raft_consensus.cc:2379] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:41.004664 4581 consensus_queue.cc:1046] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:40.963177 4008 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:40.986107 4759 consensus_queue.cc:1046] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:41.002372 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:41.025342 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:40.934463 4538 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.022851 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:40.958674 4644 consensus_queue.cc:1046] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260806 04:01:41.023839 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.003726 4571 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:41.055537 1136 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:41.056619 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 71701.1us
I20260806 04:01:41.057708 32615 raft_consensus.cc:2396] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:01:41.057729 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.060122 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:41.064044 32605 raft_consensus.cc:1226] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:41.062034 1137 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:41.065418 32741 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:41.064833 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:41.066710 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.021328 4008 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:41.042213 4182 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:41.066581 4755 log.cc:924] Time spent T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20260806 04:01:41.069407 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 109762us
I20260806 04:01:41.067950 32610 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:41.073376 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.070863 4759 raft_consensus.cc:3032] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:41.021515 4326 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260806 04:01:41.065150 3790 raft_consensus.cc:3032] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:41.087419 32604 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:41.089184 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:41.095674 4762 log.cc:924] Time spent T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.094s user 0.000s sys 0.002s
I20260806 04:01:41.060277 4625 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:41.096995 4414 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:41.121155 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 59870.7us
I20260806 04:01:41.072214 4008 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:41.026427 4544 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.124946 32473 raft_consensus.cc:1226] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:01:41.061964 4571 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.073369 4182 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.080046 4105 raft_consensus.cc:2726] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:41.142572 32478 raft_consensus.cc:1226] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:41.087213 4439 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 04:01:41.141640 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 53976.8us
W20260806 04:01:41.144966 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 86651.3us
I20260806 04:01:41.145535 32612 raft_consensus.cc:1226] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 04:01:41.086755 4759 raft_consensus.cc:729] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:41.146637 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 59058.9us
I20260806 04:01:41.147640 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.148440 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.202953 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.149436 32469 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:41.153644 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 780) took 2169 ms. Trace:
W20260806 04:01:41.156276 4751 log.cc:924] Time spent T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.153s user 0.003s sys 0.000s
I20260806 04:01:41.099665 3790 raft_consensus.cc:729] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:41.199568 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:41.204185 32746 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:41.207170 32739 raft_consensus.cc:2379] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 8: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:41.203996 1190 rpcz_store.cc:276] 0806 04:01:38.984165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.604867 (+1620702us) service_pool.cc:227] Handling call
0806 04:01:40.606256 (+ 1389us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:40.606807 (+ 551us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:40.660093 (+ 53286us) log.cc:842] Serialized 14 byte log entry
0806 04:01:40.749165 (+ 89072us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:40.765032 (+ 15867us) log.cc:842] Serialized 26 byte log entry
0806 04:01:40.765158 (+ 126us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:40.765184 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:40.765227 (+ 43us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:41.153440 (+388213us) raft_consensus.cc:1671] finished
0806 04:01:41.153466 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:41.153583 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":67110,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":738,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":8360,"thread_start_us":119655,"threads_started":2,"wal-append.queue_time_us":146310}
I20260806 04:01:41.216856 1186 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:41.164438 4439 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:41.229121 32746 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
W20260806 04:01:41.220429 4769 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260806 04:01:41.224218 3790 consensus_queue.cc:260] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.224736 4266 consensus_queue.cc:1046] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260806 04:01:41.226497 4768 log.cc:924] Time spent T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260806 04:01:41.227059 1182 raft_consensus.cc:2932] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:01:41.219906 4764 log.cc:924] Time spent T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260806 04:01:41.233107 32608 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
W20260806 04:01:41.211426 4766 log.cc:924] Time spent T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.069s user 0.000s sys 0.002s
I20260806 04:01:41.240906 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.235855 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:41.143973 4644 consensus_queue.cc:1046] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260806 04:01:41.227555 4460 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.240525 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:41.241252 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.258458 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:41.232190 4439 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:41.267004 4771 consensus_queue.cc:1046] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:41.260552 1190 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:41.264336 32608 raft_consensus.cc:1226] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:41.265818 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.266813 32748 raft_consensus.cc:2396] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 04:01:41.231935 4538 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.269873 32740 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:41.273797 32743 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:41.274832 32614 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
W20260806 04:01:41.274758 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 51660.4us
I20260806 04:01:41.234161 4759 consensus_queue.cc:260] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.332304 4771 consensus_queue.cc:1046] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:41.332690 32615 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
W20260806 04:01:41.333966 4767 log.cc:924] Time spent T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20260806 04:01:41.335685 4644 raft_consensus.cc:3032] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:41.336092 4644 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:41.307037 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.323050 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:41.310549 32743 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
W20260806 04:01:41.312299 4778 log.cc:924] Time spent T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260806 04:01:41.312863 4008 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:41.314232 4182 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:41.314844 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.250473 4326 consensus_queue.cc:1046] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260806 04:01:41.234186 4638 consensus_queue.cc:1046] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20260806 04:01:41.318328 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:41.316181 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 105431us
I20260806 04:01:41.306250 32738 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:41.320489 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:41.324805 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:41.357363 32473 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:41.300897 4560 raft_consensus.cc:3032] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:41.322335 3790 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:41.321725 4544 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.337015 4644 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:41.404381 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 83883us
I20260806 04:01:41.494611 32741 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:41.415238 32749 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:41.439905 32553 leader_election.cc:398] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8.
I20260806 04:01:41.508605 32553 leader_election.cc:401] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 04:01:41.440378 32615 raft_consensus.cc:1226] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 04:01:41.479867 32748 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:01:41.382391 32478 raft_consensus.cc:1226] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:41.412379 32739 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 04:01:41.521967 32738 raft_consensus.cc:2454] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:41.401645 1137 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:41.525733 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 99570.3us
W20260806 04:01:41.527527 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 314847us
W20260806 04:01:41.530860 4778 log.cc:924] Time spent T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.110s user 0.003s sys 0.006s
I20260806 04:01:41.371097 4625 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:41.401199 4770 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:41.395828 4571 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.420938 4182 consensus_queue.cc:1046] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.116s
W20260806 04:01:41.419909 4486 consensus_meta.cc:287] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.512s user 0.004s sys 0.006s
I20260806 04:01:41.537914 4486 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.539834 4787 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8.
I20260806 04:01:41.540426 4486 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.540810 32737 raft_consensus.cc:2379] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:41.462368 4671 consensus_meta.cc:287] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.563s user 0.007s sys 0.006s
I20260806 04:01:41.369486 4771 consensus_queue.cc:1225] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Peer d4a238f055a44f459c62c224daa4d541 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 04:01:41.382174 4326 consensus_queue.cc:1225] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer d4a238f055a44f459c62c224daa4d541 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 04:01:41.400933 4647 raft_consensus.cc:3032] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:41.536196 4544 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:41.497074 4637 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:41.400671 4784 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.136s
I20260806 04:01:41.536842 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:41.536965 4745 raft_consensus.cc:3032] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Stepping down as leader of term 7
W20260806 04:01:41.454254 4679 consensus_meta.cc:287] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.577s user 0.008s sys 0.002s
I20260806 04:01:41.563387 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:41.566427 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:41.532495 4779 log.cc:924] Time spent T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.134s user 0.003s sys 0.010s
W20260806 04:01:41.531531 4775 log.cc:924] Time spent T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.266s user 0.000s sys 0.021s
I20260806 04:01:41.543831 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:41.405442 4460 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.535826 4770 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.547297 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.548738 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.574560 1186 raft_consensus.cc:2379] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 7: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:41.506736 4560 raft_consensus.cc:729] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:41.398129 4008 leader_election.cc:288] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:41.556254 4644 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:41.556532 1136 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:41.557246 4007 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.565s user 0.004s sys 0.006s
I20260806 04:01:41.556869 4544 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.558528 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:41.551939 4647 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:41.553987 4745 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:41.560763 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.561090 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.561846 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.562659 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:41.562935 32473 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:41.564370 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.585665 1192 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 7: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:41.572485 32466 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:41.576411 1136 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:41.578014 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.553802 4105 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:41.582688 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:41.583667 32610 raft_consensus.cc:1226] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:01:41.588097 32683 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:41.563723 4439 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.592792 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.576591 4264 raft_consensus.cc:3032] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 04:01:41.598670 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 197094us
I20260806 04:01:41.573908 4656 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:41.602185 1137 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:41.578385 4266 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:41.599017 32553 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:41.607707 32604 raft_consensus.cc:1226] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:41.609064 1136 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:41.604103 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.598876 4264 raft_consensus.cc:729] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:41.611312 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:41.611720 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.616938 1186 raft_consensus.cc:2379] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:41.611972 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:41.552485 4671 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.581712 4008 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:41.619669 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.578032 4326 consensus_queue.cc:1046] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260806 04:01:41.621284 32602 raft_consensus.cc:2396] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:01:41.621505 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:41.622109 4797 consensus_queue.cc:1046] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:41.622553 32428 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:41.622295 4264 consensus_queue.cc:260] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.623281 4414 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260806 04:01:41.624027 4763 log.cc:924] Time spent T 00000000000000000000000000000000 P a4f6b68aa31d452e8d84505ace50b57d: Append to log took a long time: real 0.322s user 0.007s sys 0.007s
W20260806 04:01:41.624605 4772 log.cc:924] Time spent T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.324s user 0.009s sys 0.012s
W20260806 04:01:41.647862 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 48940us
I20260806 04:01:41.649341 32553 leader_election.cc:398] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8.
I20260806 04:01:41.637916 4656 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.650301 32553 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.651039 4746 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:41.625911 4439 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:41.611580 4266 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:41.626750 4773 log.cc:924] Time spent T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.358s user 0.002s sys 0.000s
I20260806 04:01:41.599112 4139 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:41.611568 4105 raft_consensus.cc:686] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:41.632871 32614 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:41.618848 1183 raft_consensus.cc:2379] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:41.633164 4460 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260806 04:01:41.634032 4647 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.636049 1192 raft_consensus.cc:1226] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:41.637269 32608 raft_consensus.cc:2932] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:41.636060 32478 raft_consensus.cc:1226] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:41.645087 4560 consensus_queue.cc:260] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.556583 4637 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.563441 4679 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.637112 32610 raft_consensus.cc:2379] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 8: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:41.581497 4784 consensus_queue.cc:1046] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.053s
I20260806 04:01:41.694763 4481 raft_consensus.cc:3032] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:41.654688 4105 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.695263 4481 raft_consensus.cc:729] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:41.696224 32474 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:41.697081 4679 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.698112 32478 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:01:41.624485 4638 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:41.640246 4523 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:41.625309 4771 consensus_queue.cc:1046] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:41.651489 32553 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.651216 4656 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.654062 4542 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:01:41.651999 4266 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:41.661652 4760 raft_consensus.cc:3032] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:41.722272 4760 raft_consensus.cc:729] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:41.722831 4486 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:41.723279 4760 consensus_queue.cc:260] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:41.724962 4760 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:41.661027 4759 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:41.661594 4745 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.726620 4625 consensus_queue.cc:1046] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:01:41.727164 4745 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:01:41.727671 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 76251.4us
I20260806 04:01:41.661127 4770 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.728994 32553 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.730180 32553 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.731356 4789 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:41.731734 4789 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:41.651824 4439 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:41.696223 4481 consensus_queue.cc:260] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.732106 4789 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:41.732836 4481 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:41.697919 4091 consensus_queue.cc:1046] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260806 04:01:41.733043 4782 raft_consensus.cc:2781] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:41.733953 4091 raft_consensus.cc:3032] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:41.734368 4091 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:41.733903 4782 raft_consensus.cc:686] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:41.735208 4091 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:41.624528 4800 log.cc:924] Time spent T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260806 04:01:41.735538 4782 consensus_queue.cc:237] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:41.736385 4523 consensus_queue.cc:1046] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:41.737259 4523 raft_consensus.cc:3032] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:41.737564 4523 raft_consensus.cc:729] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:41.738375 4523 consensus_queue.cc:260] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.700098 32614 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
W20260806 04:01:41.702687 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 67222.1us
I20260806 04:01:41.706724 4784 raft_consensus.cc:3032] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:41.666739 4560 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:41.714583 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 92094.1us
I20260806 04:01:41.716897 1187 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:41.716753 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:41.744203 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.694396 4264 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:41.760994 4638 raft_consensus.cc:686] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:41.762161 4638 consensus_queue.cc:237] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.662111 4647 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:41.660281 4544 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.655000 4671 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.713395 4414 raft_consensus.cc:3032] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:41.745229 1186 raft_consensus.cc:2354] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4.
I20260806 04:01:41.733026 4746 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:41.778481 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:41.777997 32740 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:01:41.719597 4007 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:41.778218 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 46241.8us
I20260806 04:01:41.794291 32748 raft_consensus.cc:1226] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:41.779429 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.781908 4656 raft_consensus.cc:3032] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:41.784291 3790 consensus_queue.cc:1046] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:41.778007 4439 raft_consensus.cc:686] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:41.785035 4544 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:01:41.785892 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.786111 1186 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:41.786229 32614 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 04:01:41.777655 32682 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.777391 4139 raft_consensus.cc:686] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:41.792341 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:41.804126 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.805894 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.770550 4523 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:41.758711 4784 raft_consensus.cc:729] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:41.818783 1190 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260806 04:01:41.777527 4414 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:41.807087 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 204967us
I20260806 04:01:41.811478 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.814877 32609 raft_consensus.cc:2379] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:41.799494 4656 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:41.815193 32602 raft_consensus.cc:2379] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:41.816519 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.828584 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.830080 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.819691 32682 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.816637 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.806746 4789 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.832902 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.817265 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.827899 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.837188 32614 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 7: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:41.819542 4266 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.820292 4671 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:41.839995 4266 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:41.802040 4091 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:41.840029 1137 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:41.843071 32682 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.855509 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:41.848605 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.856499 32614 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:41.840621 4812 consensus_queue.cc:1046] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260806 04:01:41.825035 4139 consensus_queue.cc:237] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:41.867589 4812 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:41.868284 4812 raft_consensus.cc:3032] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:41.868597 4812 raft_consensus.cc:729] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:41.870013 4182 consensus_queue.cc:1046] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:41.871181 32682 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.871546 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.869802 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:41.875633 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 45646.1us
I20260806 04:01:41.876259 32682 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.877385 32551 leader_election.cc:398] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4.
I20260806 04:01:41.877715 32682 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:41.877724 32551 leader_election.cc:401] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 04:01:41.881074 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.838037 4571 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:41.880698 32614 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:41.882947 1137 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:41.925015 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:41.925122 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.926702 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.927572 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.928259 4816 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:41.928798 4816 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:41.930882 4816 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:41.931412 4816 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:41.828989 4656 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.815323 4679 consensus_queue.cc:1046] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260806 04:01:41.932245 4581 raft_consensus.cc:3032] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:41.932540 4656 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:41.932678 4581 raft_consensus.cc:729] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:41.933652 4581 consensus_queue.cc:260] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.882215 4486 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:41.881955 4481 raft_consensus.cc:3032] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:41.936874 4481 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:41.937659 4808 log.cc:924] Time spent T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 04:01:41.939226 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.939757 32614 raft_consensus.cc:1226] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:41.924978 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.940313 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:41.936061 1185 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:01:41.927896 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.878741 4396 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:41.949841 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.952375 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:41.955096 32615 raft_consensus.cc:2396] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:01:41.957613 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.960047 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:41.960747 32602 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:41.881189 4671 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:41.884302 4007 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4.
I20260806 04:01:41.884305 4625 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:41.964051 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:41.964483 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:41.962175 32466 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:41.985091 32683 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:41.938817 4542 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:41.926599 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:41.938656 4659 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:41.980192 32609 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:41.981882 1185 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:01:41.987686 1190 raft_consensus.cc:1226] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:41.931516 4789 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:41.954798 4266 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:41.869450 4812 consensus_queue.cc:260] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.884166 4644 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20260806 04:01:41.836866 4439 consensus_queue.cc:237] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.020519 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:42.021517 1184 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 4: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:42.022001 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:42.028726 4821 log.cc:924] Time spent T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260806 04:01:41.937808 4481 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.029987 4481 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:42.033586 1188 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57c0e905630e43668379f2f95684775f"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:42.034413 1188 raft_consensus.cc:595] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:42.034874 1188 raft_consensus.cc:603] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:42.036521 32610 raft_consensus.cc:2396] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:01:41.975102 4396 raft_consensus.cc:686] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:41.818805 4784 consensus_queue.cc:260] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:41.950477 4746 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.059s
I20260806 04:01:41.991622 4637 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:42.049021 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 194162us
I20260806 04:01:42.006723 4625 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:42.065752 32602 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:42.067384 4625 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:41.993108 4647 consensus_queue.cc:1046] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260806 04:01:42.035619 4787 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:42.070641 1136 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:42.076272 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 198617us
I20260806 04:01:42.071079 32612 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:42.080741 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:42.061075 4799 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.074527 4625 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:42.079638 4400 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:42.103549 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 139445us
I20260806 04:01:42.082499 4659 raft_consensus.cc:2781] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:42.110530 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
W20260806 04:01:42.137279 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 172870us
I20260806 04:01:42.140005 32745 raft_consensus.cc:1261] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:01:42.067507 4806 consensus_queue.cc:1046] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.063s
I20260806 04:01:42.189402 32682 leader_election.cc:302] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:42.200428 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 116597us
I20260806 04:01:42.201287 32466 raft_consensus.cc:1226] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:42.200816 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.202649 32475 raft_consensus.cc:3037] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:42.202538 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.193439 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.205902 32739 raft_consensus.cc:2396] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:01:42.196458 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.148484 4659 raft_consensus.cc:686] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:42.212556 32608 raft_consensus.cc:1226] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:42.213967 32474 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:42.217957 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:42.218640 4818 log.cc:924] Time spent T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20260806 04:01:42.192960 32748 raft_consensus.cc:2932] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:42.214412 4656 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:42.224367 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.225134 32749 raft_consensus.cc:2379] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 3: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:42.225771 4659 consensus_queue.cc:237] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.190536 4400 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.098759 4799 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:42.213905 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.252573 32738 raft_consensus.cc:2379] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:42.232987 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:42.236781 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 48671.8us
W20260806 04:01:42.250203 4824 log.cc:924] Time spent T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.121s user 0.000s sys 0.002s
I20260806 04:01:42.213840 4139 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.148257 4396 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.209071 32427 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:42.277002 32739 raft_consensus.cc:2379] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:42.280264 4644 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:42.280946 4644 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:42.281484 32553 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.282605 4836 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:42.284466 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 82065us
I20260806 04:01:42.283464 4644 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:42.288021 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.282696 32553 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.293255 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:42.295584 4264 consensus_meta.cc:287] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.521s user 0.006s sys 0.001s
W20260806 04:01:42.296217 4838 log.cc:924] Time spent T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.051s user 0.002s sys 0.000s
W20260806 04:01:42.297319 4832 log.cc:924] Time spent T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.087s user 0.002s sys 0.000s
I20260806 04:01:42.298931 4799 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:42.299491 4799 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:42.299806 4799 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:42.300343 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.287806 4139 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:42.302698 4139 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:42.305029 4783 raft_consensus.cc:3032] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:42.305500 4783 raft_consensus.cc:729] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:42.306411 4783 consensus_queue.cc:260] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.288333 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:42.313911 32745 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260806 04:01:42.312680 4486 consensus_queue.cc:1046] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
W20260806 04:01:42.289068 4759 consensus_meta.cc:287] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.563s user 0.007s sys 0.003s
I20260806 04:01:42.318902 32474 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:42.332587 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.318363 32749 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:42.321444 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:42.289906 4542 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:42.305595 4571 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:42.313298 4783 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:42.336391 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:42.312961 4799 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:42.320034 4486 raft_consensus.cc:3032] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:42.339007 32746 raft_consensus.cc:1226] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260806 04:01:42.319324 4841 log.cc:924] Time spent T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.123s user 0.000s sys 0.002s
I20260806 04:01:42.339560 32477 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260806 04:01:42.340552 32427 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.340868 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:42.342974 32741 raft_consensus.cc:1226] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:42.344463 32738 raft_consensus.cc:1226] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:42.338503 4746 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.317365 4659 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:42.343169 32682 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.321959 4644 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:42.346011 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:42.343844 32477 pending_rounds.cc:76] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 5
I20260806 04:01:42.354948 32475 raft_consensus.cc:2454] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:42.358747 32477 pending_rounds.cc:97] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260806 04:01:42.357271 1137 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.346225 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.357203 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:42.358063 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 219994us
I20260806 04:01:42.359043 32477 raft_consensus.cc:2866] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:42.356395 4105 consensus_queue.cc:1046] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.004s
I20260806 04:01:42.360586 4542 raft_consensus.cc:686] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:42.370103 32682 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:42.375169 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 92554.1us
I20260806 04:01:42.342711 4486 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:42.378346 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.379644 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.381659 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.376771 32553 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.384671 32738 raft_consensus.cc:1226] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:42.384929 32466 raft_consensus.cc:2354] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4.
I20260806 04:01:42.335135 4625 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:42.386360 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.375211 4759 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 04:01:42.392058 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 51600.1us
I20260806 04:01:42.392840 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.394109 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.377699 4571 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:42.395426 32427 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.409363 1189 raft_consensus.cc:2396] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:01:42.400313 32553 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.417505 1137 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.390682 4816 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.424729 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.431378 32474 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:42.425833 4523 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:42.426901 4679 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:42.368769 4659 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.398288 4671 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:42.393618 4105 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:42.436062 32746 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:42.356711 4746 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:42.440805 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 45476.9us
I20260806 04:01:42.454533 32427 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.394481 4656 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 04:01:42.392803 4789 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:42.409049 4847 raft_consensus.cc:3032] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:42.447851 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.398211 4830 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:42.427239 32553 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.463790 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.461153 4759 raft_consensus.cc:686] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:42.439030 4439 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:42.395375 4828 consensus_queue.cc:1046] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
W20260806 04:01:42.476370 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 82661.1us
I20260806 04:01:42.480895 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.393545 4855 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260806 04:01:42.465919 4759 consensus_queue.cc:237] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:42.459833 4847 raft_consensus.cc:729] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:42.516367 32739 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:42.516848 32475 raft_consensus.cc:1226] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 9. Current term is 10. Ops: []
I20260806 04:01:42.537181 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:42.537905 32473 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:42.513949 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.531205 32427 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.466984 4523 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:42.445909 4816 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.473730 4745 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:42.556465 4828 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:42.562219 4853 consensus_queue.cc:1046] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260806 04:01:42.564469 4091 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.565053 4091 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.567027 4091 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:42.567848 4625 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:42.568552 4890 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.569087 4890 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.569541 4266 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.570115 4266 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.571197 4890 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:42.459578 4486 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:42.526703 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 67090.8us
I20260806 04:01:42.477137 4400 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:42.577710 4400 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:42.577987 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.528111 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.579322 32738 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
I20260806 04:01:42.580001 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:42.568575 1192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6023572bff444c85842d7b3351debf87"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:42.581281 1192 raft_consensus.cc:595] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.476919 4542 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.581594 1192 raft_consensus.cc:603] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:42.578334 4400 consensus_queue.cc:1046] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.006s
W20260806 04:01:42.590265 4867 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 04:01:42.476718 4787 consensus_queue.cc:1046] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.115s
I20260806 04:01:42.591812 4787 consensus_queue.cc:1225] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer f7c202b562ce482db283463d7f02e136 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:01:42.592288 4538 raft_consensus.cc:3032] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:42.592640 4538 raft_consensus.cc:729] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:42.593679 4884 raft_consensus.cc:3032] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:42.593583 4538 consensus_queue.cc:260] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:42.594107 4884 raft_consensus.cc:729] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:42.594892 4538 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:42.595135 4884 consensus_queue.cc:260] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.515343 4679 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:42.665158 4105 raft_consensus.cc:686] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:42.446978 4799 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.671002 32469 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:42.477792 4836 consensus_queue.cc:1046] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260806 04:01:42.578750 32739 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:42.596016 4488 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.596771 4659 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:42.597666 4847 consensus_queue.cc:260] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.599236 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:42.677676 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:42.609468 4644 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:42.611284 4886 raft_consensus.cc:3032] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 04:01:42.537231 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 63914.5us
I20260806 04:01:42.666960 32467 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:01:42.729920 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:42.669000 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.669176 4746 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:42.666169 4266 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:42.670014 32473 raft_consensus.cc:1261] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:42.669000 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:42.672864 1137 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:42.541146 32682 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.678221 32474 raft_consensus.cc:1226] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:42.538739 4396 consensus_queue.cc:1046] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.143s
I20260806 04:01:42.678349 32471 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 9.
W20260806 04:01:42.742008 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 69238.1us
W20260806 04:01:42.743206 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 202137us
I20260806 04:01:42.743942 32682 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
W20260806 04:01:42.743942 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 212839us
I20260806 04:01:42.731073 4886 raft_consensus.cc:729] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:42.674584 4679 raft_consensus.cc:686] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:42.679900 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.753819 1137 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.754005 32738 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 9.
I20260806 04:01:42.678608 4523 consensus_queue.cc:1046] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.077s
I20260806 04:01:42.755599 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:42.755942 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:42.756247 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:42.756888 32610 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:42.748836 1136 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.758757 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:42.756786 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:42.759804 32615 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:42.749632 32683 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:42.750875 32427 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.757879 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:42.759353 32605 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:42.768903 32614 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
W20260806 04:01:42.752198 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 211382us
I20260806 04:01:42.771454 32427 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.773748 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:42.774137 32475 raft_consensus.cc:1226] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:42.774362 1190 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:42.774981 1190 raft_consensus.cc:595] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.775398 1190 raft_consensus.cc:603] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:42.775022 4803 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260806 04:01:42.776279 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:42.776858 1190 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
W20260806 04:01:42.777206 32532 heartbeater.cc:644] Failed to heartbeat to 127.29.237.190:40089 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.29.237.190:40089 timed out after 15.000s (SENT)
I20260806 04:01:42.778275 4853 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:42.778962 4853 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:42.779594 4853 consensus_queue.cc:1225] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer ae0b3e43e7a64f789e51018fd5518d23 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 04:01:42.780305 4891 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.780830 4891 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.781301 4816 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:42.744225 4266 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.782207 4266 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.782855 4891 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:42.783492 4877 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:42.784415 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:42.679563 4488 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.786005 4886 consensus_queue.cc:260] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:42.787675 4488 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:42.788260 4488 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:42.788743 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:42.788822 4488 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:42.789448 1188 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
I20260806 04:01:42.790916 32551 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:42.747574 4847 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:42.752259 4659 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:42.792513 4659 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:42.792850 4659 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:42.797143 1188 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:42.797652 4396 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:42.798062 4396 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:42.798429 4396 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:42.763412 4883 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:42.808454 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:42.810945 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:42.812300 32749 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:01:42.812551 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.751782 4875 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:42.816956 32467 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 04:01:42.817844 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 46488.3us
I20260806 04:01:42.822147 1137 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.824431 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:42.807607 4542 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:42.790863 4488 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:42.823400 32427 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.808595 4266 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:42.834399 1189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:42.841315 1189 raft_consensus.cc:595] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:01:42.816412 4860 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:42.839557 32428 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:42.819736 4803 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:42.851451 32428 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:42.849790 32605 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:42.824395 4892 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:42.825712 4656 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:42.860405 4656 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:42.860857 4656 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:42.851141 4803 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:42.861845 4803 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:42.850925 4770 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.827898 4883 raft_consensus.cc:686] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:42.819344 4847 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260806 04:01:42.863536 4770 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.824684 4783 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:42.864401 4883 consensus_queue.cc:237] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.898502 32474 pending_rounds.cc:76] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 3
I20260806 04:01:42.899493 32474 pending_rounds.cc:97] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260806 04:01:42.899732 32474 raft_consensus.cc:2866] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 04:01:42.886363 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 63034.7us
I20260806 04:01:42.901404 32466 raft_consensus.cc:1226] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 04:01:42.901796 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:42.898420 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 47070.2us
W20260806 04:01:42.907541 32478 consensus_meta.cc:287] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.574s user 0.009s sys 0.008s
I20260806 04:01:42.907996 32478 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:42.907895 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:42.908674 32612 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 1945) took 1361 ms (client timeout 1535 ms). Trace:
W20260806 04:01:42.909044 32612 rpcz_store.cc:269] 0806 04:01:41.546904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:41.547055 (+ 151us) service_pool.cc:227] Handling call
0806 04:01:42.908527 (+1361472us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b5400086954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:42.908620 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":616736512}
I20260806 04:01:42.914815 32604 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 04:01:42.932881 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:42.936518 1185 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:01:42.885281 4746 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:42.878492 4544 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:42.885502 4836 raft_consensus.cc:3032] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:42.877568 4523 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.064s
I20260806 04:01:42.941426 4856 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
I20260806 04:01:42.944936 4414 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:42.913497 4326 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:42.860042 4679 consensus_queue.cc:237] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:42.891682 4860 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:42.912606 4855 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:42.946967 4799 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:42.941359 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:43.043437 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.975565 32749 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:42.977643 32737 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:01:42.830361 4105 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.035514 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:42.950843 4544 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.045405 32743 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
W20260806 04:01:43.056183 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 306607us
I20260806 04:01:42.951133 4836 raft_consensus.cc:729] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:43.061132 32427 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:43.061154 1192 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260806 04:01:43.062021 1192 pending_rounds.cc:76] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: Aborting all ops after (but not including) 6
I20260806 04:01:43.062332 1192 pending_rounds.cc:97] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260806 04:01:43.062611 1192 raft_consensus.cc:2866] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:43.035514 4855 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:42.907918 4892 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.035907 4488 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.044240 4414 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:43.075856 32427 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:43.071220 32428 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:43.078887 4789 consensus_meta.cc:287] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.618s user 0.004s sys 0.010s
I20260806 04:01:43.080109 4875 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.081600 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.086438 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.087316 4784 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:43.065343 4869 log.cc:924] Time spent T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260806 04:01:43.040954 4326 raft_consensus.cc:686] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:43.061952 4544 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Advancing to term 10
W20260806 04:01:43.167333 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 91555.7us
I20260806 04:01:43.205729 32427 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:42.968499 4746 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.063730 1183 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
W20260806 04:01:43.196713 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 125581us
I20260806 04:01:43.093729 4855 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:43.202970 32466 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:01:43.166811 4789 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:43.098907 4891 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:43.099987 4812 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:43.068475 4853 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.111s
I20260806 04:01:43.097370 4770 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:43.199306 4538 consensus_meta.cc:287] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.578s user 0.007s sys 0.018s
I20260806 04:01:43.214480 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 576) took 1056 ms. Trace:
I20260806 04:01:43.226025 32603 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:43.231807 32748 rpcz_store.cc:276] 0806 04:01:42.143223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:42.143363 (+ 140us) service_pool.cc:227] Handling call
0806 04:01:42.192634 (+ 49271us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:42.192843 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:42.348418 (+155575us) log.cc:842] Serialized 14 byte log entry
0806 04:01:42.545696 (+197278us) log.cc:842] Serialized 14 byte log entry
0806 04:01:42.866718 (+321022us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b1800398380. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a32c638 00007fd16a32fb11 00007fd16a32fac9 00007fd16a32fa59 00007fd16a32fa21 00007fd16a32ed1d 00007fd173590204 00007fd173590039 00007fd16a30edc6
0806 04:01:42.889450 (+ 22732us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b1800398380. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a329155 00007fd16d24c3d0 00007fd16d25e576 00007fd16d26f0d7 00007fd16d26f058
0806 04:01:43.197522 (+308072us) log.cc:842] Serialized 14 byte log entry
0806 04:01:43.199483 (+ 1961us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:43.199535 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:43.199589 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:43.199654 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:43.199765 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11258,"raft.queue_time_us":18718,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":11393,"spinlock_wait_cycles":175319424,"thread_start_us":493170,"threads_started":2,"wal-append.queue_time_us":248470}
I20260806 04:01:43.204525 4759 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:43.237195 32609 raft_consensus.cc:1261] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:43.211586 4799 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:43.165553 4488 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.221668 4439 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.251529 32553 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.235086 4877 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:43.244033 4799 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.205440 4875 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.242280 4856 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260806 04:01:43.293495 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.238170 4914 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:43.285719 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 812) took 1962 ms. Trace:
I20260806 04:01:43.296785 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:43.276345 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 60318.2us
I20260806 04:01:43.303658 32477 rpcz_store.cc:276] 0806 04:01:41.323422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:41.323604 (+ 182us) service_pool.cc:227] Handling call
0806 04:01:41.323897 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:42.373369 (+1049472us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:43.285462 (+912093us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540009db54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:43.285638 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2325149056}
I20260806 04:01:43.255535 4921 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.258903 4806 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.311136 1137 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.316543 4879 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.317255 4879 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.319501 4879 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.327858 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.328047 4544 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.329008 32739 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:43.330549 1186 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:43.329886 4771 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:43.328332 4784 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.332044 4771 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.332237 4784 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.335458 32428 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.334235 4771 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.336210 4771 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:43.336645 4771 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.337093 4771 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:43.337147 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.340531 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:43.341178 32739 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:43.344254 4784 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:43.346566 4784 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:43.348417 4784 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.277338 4266 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.196298 4836 consensus_queue.cc:260] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.291901 4644 raft_consensus.cc:3032] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:43.274715 4799 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:43.330150 4544 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.297586 4922 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.364430 32749 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
W20260806 04:01:43.364833 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 159098us
I20260806 04:01:43.285123 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 894) took 1545 ms. Trace:
I20260806 04:01:43.277889 4488 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.291765 4892 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.376842 32427 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.353451 4182 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:43.353143 4830 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.257267 4439 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.354278 4915 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:43.381248 32427 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:43.382324 4932 log.cc:924] Time spent T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260806 04:01:43.374473 4266 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.411154 4266 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.293844 4877 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:43.392026 4656 consensus_meta.cc:287] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.531s user 0.004s sys 0.007s
I20260806 04:01:43.413230 4656 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.414072 4877 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.354444 4921 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.415237 4656 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:43.422701 4396 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.624s user 0.006s sys 0.012s
I20260806 04:01:43.421818 4784 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:43.423151 4396 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.398571 4760 consensus_queue.cc:1046] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:43.424700 4849 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.425217 4849 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.428529 32739 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 10.
I20260806 04:01:43.429734 4855 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.432426 4830 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.399605 4922 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.436121 4922 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.357426 4789 consensus_queue.cc:1046] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260806 04:01:43.385913 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 923) took 1511 ms. Trace:
I20260806 04:01:43.386942 4947 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:43.388136 4886 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:43.440531 1187 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:43.398885 4746 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.291903 4894 consensus_queue.cc:1035] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.303s
I20260806 04:01:43.392299 4544 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 04:01:43.393674 4875 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:43.396683 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:43.447791 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 196409us
I20260806 04:01:43.373461 4644 raft_consensus.cc:729] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:43.273488 4326 consensus_queue.cc:237] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.437780 1186 rpcz_store.cc:276] 0806 04:01:41.874253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:41.927318 (+ 53065us) service_pool.cc:227] Handling call
0806 04:01:43.385704 (+1458386us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540079d1d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:43.385822 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3184937856}
I20260806 04:01:43.431831 4855 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.440732 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.279892 4659 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:43.441310 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 60160.8us
W20260806 04:01:43.390087 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 627933us
I20260806 04:01:43.445393 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:43.491112 32745 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:43.449169 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.448838 32477 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:43.362440 4806 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.495353 4806 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.506520 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:43.507189 32477 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:43.507073 32683 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.509708 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:43.486793 32428 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.443862 4914 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 04:01:43.510677 1186 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:43.484634 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
W20260806 04:01:43.513127 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 174369us
I20260806 04:01:43.482718 4784 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.516858 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.517486 32745 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:43.517961 32478 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:43.487643 4875 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:43.518260 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 69171us
I20260806 04:01:43.525617 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:43.525884 32553 leader_election.cc:398] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4.
I20260806 04:01:43.526820 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.522951 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.523331 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:43.521273 32428 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.528308 32605 raft_consensus.cc:2354] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6.
I20260806 04:01:43.528568 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:43.529920 1137 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:43.530766 4638 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:43.531155 4638 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.531522 4638 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:43.530774 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.532244 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.532433 32749 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:43.532939 32738 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:43.534912 1137 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.538563 1137 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.539281 32427 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.515162 4571 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:43.545684 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.546824 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.546890 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.528923 4812 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:43.549388 32615 raft_consensus.cc:1226] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:43.380270 1182 rpcz_store.cc:276] 0806 04:01:41.739548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:41.743526 (+ 3978us) service_pool.cc:227] Handling call
0806 04:01:41.802629 (+ 59103us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:41.836443 (+ 33814us) raft_consensus.cc:1520] Early marking committed up to index 11
0806 04:01:41.950192 (+113749us) log.cc:842] Serialized 14 byte log entry
0806 04:01:43.284553 (+1334361us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b1800480580. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16cee382c 00007fd16a30e9dc 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a329155 00007fd16d24c3d0 00007fd16d25e576 00007fd16d26f0d7
0806 04:01:43.284677 (+ 124us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:43.284735 (+ 58us) raft_consensus.cc:1631] Marking committed up to 11
0806 04:01:43.284795 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:43.284882 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:43.285031 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9777,"prepare.queue_time_us":77862,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":58996,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":19139,"spinlock_wait_cycles":22995200,"thread_start_us":188315,"threads_started":2,"wal-append.queue_time_us":177392,"wal-append.run_cpu_time_us":3554,"wal-append.run_wall_time_us":1166911}
I20260806 04:01:43.521894 4789 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:43.546384 32605 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:43.553294 32615 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:43.529065 4877 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 04:01:43.637214 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 92212.8us
I20260806 04:01:43.529253 4914 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:43.643004 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 61739.9us
I20260806 04:01:43.535634 1184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:43.553329 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:43.639218 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:43.634352 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 87610.9us
I20260806 04:01:43.652799 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.653440 32605 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:43.550195 4647 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:43.653896 32553 leader_election.cc:302] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.654331 4647 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.654763 4647 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:43.655148 32553 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:43.542587 4875 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:43.528740 4947 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:01:43.558413 32471 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 10: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:43.602684 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:43.641826 4939 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:43.554735 4745 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.574332 4770 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:43.541191 4921 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.547104 4892 raft_consensus.cc:2781] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:43.676992 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.543403 4571 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.521273 4784 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.573359 4486 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:43.552963 4940 log.cc:924] Time spent T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 04:01:43.664969 32608 raft_consensus.cc:2379] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 8: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:43.550276 4855 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:43.642463 4266 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.550441 4950 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:01:43.547287 4782 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:43.549476 4812 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.708715 4812 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:43.712446 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.713178 32608 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:43.714109 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.714707 32608 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:43.715781 4266 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.718106 4266 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:43.718376 4784 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.715597 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:43.720113 32608 raft_consensus.cc:3037] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:43.678737 4892 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.721335 4892 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:43.560873 32602 raft_consensus.cc:1261] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:43.541482 4182 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.118s
I20260806 04:01:43.575613 4879 raft_consensus.cc:3032] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:43.723428 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.724208 4939 consensus_queue.cc:1046] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 9.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260806 04:01:43.671638 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.677080 4745 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:43.725942 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.726852 32610 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 04:01:43.727125 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.727725 4745 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.575376 4912 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:43.572710 4956 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:43.784758 4956 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.561759 4396 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.787238 4956 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.788069 4912 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.790169 4912 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.563690 4659 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.677701 32605 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 3.
I20260806 04:01:43.524377 4644 consensus_queue.cc:260] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.665160 4746 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:43.684842 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.679589 4921 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.685408 32427 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.561277 4799 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.799715 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.692219 4842 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:43.659089 4914 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.693030 4955 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.565771 4952 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.130s
I20260806 04:01:43.696233 4091 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:43.699179 4875 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:43.667795 1184 raft_consensus.cc:595] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Received request to transfer leadership to TS f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.552263 4936 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.126s
I20260806 04:01:43.673623 4488 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.852859 4488 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.854090 4856 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.854527 4856 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.855001 4488 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.668789 32738 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:43.856503 4856 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:43.857573 32746 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:01:43.857800 32603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42618 (request call id 694) took 1278 ms. Trace:
I20260806 04:01:43.682615 4789 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:43.678431 4947 raft_consensus.cc:686] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:43.858548 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.723596 4879 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:43.804592 4782 raft_consensus.cc:686] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:43.576218 4915 raft_consensus.cc:3032] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:43.861021 4915 raft_consensus.cc:729] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:43.861323 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1958) took 2271 ms. Trace:
I20260806 04:01:43.859481 4947 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.861429 4782 consensus_queue.cc:237] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.860705 4879 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.861757 4915 consensus_queue.cc:260] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.864030 4915 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:43.824949 4921 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:43.866122 32738 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:43.679448 4571 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:43.679138 4771 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.868368 4799 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.869051 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:43.869635 32738 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:43.869974 4771 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:43.701318 4928 log.cc:924] Time spent T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.413s user 0.003s sys 0.000s
I20260806 04:01:43.728062 32615 raft_consensus.cc:2396] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:01:43.858142 32603 rpcz_store.cc:276] 0806 04:01:42.579542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:42.579723 (+ 181us) service_pool.cc:227] Handling call
0806 04:01:43.857633 (+1277910us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b5400306454. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:43.857745 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1483970944}
I20260806 04:01:43.739194 4182 raft_consensus.cc:979] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541: : Instructing follower f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:01:43.861743 32604 rpcz_store.cc:276] 0806 04:01:41.591561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:41.622964 (+ 31403us) service_pool.cc:227] Handling call
0806 04:01:42.908650 (+1285686us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:42.915344 (+ 6694us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:43.861115 (+945771us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400086950. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:43.861245 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2956048256}
I20260806 04:01:43.872473 4182 raft_consensus.cc:1067] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Signalling peer f7c202b562ce482db283463d7f02e136 to start an election
I20260806 04:01:43.742441 4849 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.744724 32477 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
W20260806 04:01:43.745558 4942 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.456s user 0.001s sys 0.000s
W20260806 04:01:43.748376 4937 log.cc:924] Time spent T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.453s user 0.003s sys 0.000s
I20260806 04:01:43.724707 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:43.758212 4007 raft_consensus.cc:3032] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:43.765244 4523 raft_consensus.cc:3032] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 LEADER]: Stepping down as leader of term 9
W20260806 04:01:43.766491 4920 log.cc:924] Time spent T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.482s user 0.002s sys 0.000s
W20260806 04:01:43.771260 1187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 918) took 1900 ms (client timeout 1643 ms). Trace:
I20260806 04:01:43.882628 32739 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
W20260806 04:01:43.883029 1187 rpcz_store.cc:269] 0806 04:01:41.871067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:41.872981 (+ 1914us) service_pool.cc:227] Handling call
0806 04:01:43.771051 (+1898070us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b5400611054. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:43.771180 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3420413824}
I20260806 04:01:43.881197 32602 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:43.772557 32609 raft_consensus.cc:1226] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 1. Current term is 2. Ops: []
I20260806 04:01:43.773252 32608 raft_consensus.cc:2454] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
W20260806 04:01:43.667064 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 173674us
I20260806 04:01:43.795239 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.796183 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.800277 32614 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260806 04:01:43.885190 32614 pending_rounds.cc:76] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 6
I20260806 04:01:43.738560 4830 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.691162 4647 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.787725 4581 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:43.890740 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:43.891323 32748 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:01:43.891597 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 229840us
W20260806 04:01:43.815186 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 132588us
I20260806 04:01:43.815210 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.892973 32469 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:43.846670 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.843773 4460 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:43.893867 32470 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:43.729691 4638 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.852267 4914 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.872092 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 822) took 1061 ms. Trace:
I20260806 04:01:43.872732 32478 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 04:01:43.881649 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.885397 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.900377 32477 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:43.885432 32614 pending_rounds.cc:97] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260806 04:01:43.901078 32614 raft_consensus.cc:2866] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:43.886477 1137 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.902419 1137 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:43.814073 4644 raft_consensus.cc:3037] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:43.870520 4904 log.cc:924] Time spent T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.696s user 0.001s sys 0.000s
I20260806 04:01:43.883543 32683 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.910488 32738 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:43.890202 32473 raft_consensus.cc:2379] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:43.807308 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.911710 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.888818 32746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
from {username='slave'} at 127.0.0.1:33182
I20260806 04:01:43.887452 4523 raft_consensus.cc:729] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:43.882767 4007 raft_consensus.cc:729] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:43.847280 4816 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:43.912916 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.913712 32473 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:43.892529 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:43.912256 32740 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:43.873819 4784 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.845970 4659 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:43.887624 4803 consensus_meta.cc:287] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.025s user 0.003s sys 0.018s
I20260806 04:01:43.898795 32475 rpcz_store.cc:276] 0806 04:01:42.810231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:42.810720 (+ 489us) service_pool.cc:227] Handling call
0806 04:01:42.829749 (+ 19029us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:42.883400 (+ 53651us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:42.883432 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:42.884008 (+ 576us) log.cc:842] Serialized 26 byte log entry
0806 04:01:43.098389 (+214381us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:43.098424 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:43.098494 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:43.700596 (+602102us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:43.871767 (+171171us) raft_consensus.cc:1671] finished
0806 04:01:43.871788 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:43.872012 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47864,"raft.queue_time_us":54585,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":26967808,"thread_start_us":213950,"threads_started":1,"wal-append.queue_time_us":374344}
I20260806 04:01:43.907570 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.909046 32743 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260806 04:01:43.921526 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.922298 32740 raft_consensus.cc:2354] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5.
I20260806 04:01:43.866928 4921 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.912801 32746 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:01:43.926470 32746 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:43.926736 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.850174 1184 raft_consensus.cc:603] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:43.914235 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.925107 32602 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:43.915088 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.910380 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.916801 32682 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.918429 32428 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:43.931293 32748 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
W20260806 04:01:43.894181 4679 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.008s sys 0.020s
I20260806 04:01:43.932914 32553 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.933231 1137 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:43.927423 32473 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:43.936306 32553 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:43.892328 4856 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260806 04:01:43.911547 4783 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:43.938959 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.939143 32553 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:43.927811 32469 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:43.878868 4091 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.915295 4488 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.911688 4849 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:43.928717 4922 raft_consensus.cc:2781] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:43.941756 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:43.948323 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.926740 4914 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:43.942440 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:43.943499 32743 raft_consensus.cc:1226] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:43.898054 4789 consensus_queue.cc:1046] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 04:01:43.942164 1137 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
W20260806 04:01:43.932534 4679 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.009s sys 0.020s
I20260806 04:01:43.966738 32749 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:43.950294 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:43.980762 32614 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:01:43.939144 4812 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:43.920537 4936 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:43.956771 32469 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:43.951300 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.954633 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.952426 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.954228 32738 raft_consensus.cc:1203] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 3.3->[5.4-5.4] Dedup: 5.4->[]
I20260806 04:01:43.992681 32738 raft_consensus.cc:1226] T 18f3a6d6cef24470a69dbafa39b3d3e1 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 04:01:43.936827 4855 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:43.946146 4849 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.992123 32741 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:43.995675 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 825) took 1117 ms. Trace:
I20260806 04:01:43.966684 32553 leader_election.cc:398] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5.
I20260806 04:01:43.968902 32466 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:43.959023 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.936713 4264 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:43.990912 32427 leader_election.cc:398] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6.
I20260806 04:01:43.969663 32746 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.917728 4816 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:43.987434 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 45333.6us
I20260806 04:01:43.942487 4921 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:43.942858 4746 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260806 04:01:43.960630 4960 log.cc:924] Time spent T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 04:01:43.991808 32605 raft_consensus.cc:2379] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:43.992825 32740 raft_consensus.cc:2396] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 04:01:43.991801 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:43.997354 32553 leader_election.cc:401] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:43.951237 4745 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:43.990907 4812 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.999044 32427 leader_election.cc:401] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:01:44.000862 32746 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:43.968696 4914 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:43.995623 4806 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:44.004002 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:43.924335 4784 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.995286 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 04:01:43.998320 4892 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.999171 4849 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:44.006866 4091 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.008646 32614 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:43.999416 4936 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.980172 4956 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:43.927234 4659 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.014497 4745 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.016839 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:44.024185 4812 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:43.956235 4460 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.008224 4647 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:43.991407 4783 raft_consensus.cc:686] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:44.006942 4886 consensus_meta.cc:287] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.564s user 0.006s sys 0.003s
I20260806 04:01:44.023417 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.026048 32738 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 04:01:43.949384 4488 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.926059 4803 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:43.845224 4955 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.011108 4929 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:43.996048 32474 rpcz_store.cc:276] 0806 04:01:42.878310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:42.878790 (+ 480us) service_pool.cc:227] Handling call
0806 04:01:42.893716 (+ 14926us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:42.976277 (+ 82561us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:42.976312 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:43.077581 (+101269us) log.cc:842] Serialized 598 byte log entry
0806 04:01:43.265548 (+187967us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:43.265587 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:43.265657 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:43.744676 (+479019us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:43.966112 (+221436us) raft_consensus.cc:1671] finished
0806 04:01:43.966134 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:43.995604 (+ 29470us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":167837,"raft.queue_time_us":120856,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":16004,"spinlock_wait_cycles":33774336,"thread_start_us":175972,"threads_started":1,"wal-append.queue_time_us":199847}
I20260806 04:01:44.032958 32748 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 04:01:44.038895 4745 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:44.033783 32428 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:44.009902 4855 raft_consensus.cc:686] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:44.002933 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:44.008294 4007 consensus_queue.cc:260] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.018118 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:44.079025 32740 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:44.026232 4400 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6.
I20260806 04:01:44.027982 4914 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:44.058598 4460 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.083519 4488 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.028647 4806 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:44.044344 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 61373.9us
I20260806 04:01:44.031643 4746 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5.
I20260806 04:01:44.108184 4806 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:43.988114 4523 consensus_queue.cc:260] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.065239 32470 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:44.030507 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 63940.6us
I20260806 04:01:44.024879 4571 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:44.071928 4679 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election won for term 8
W20260806 04:01:44.063016 4971 log.cc:924] Time spent T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20260806 04:01:44.037647 32466 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:44.063323 4967 log.cc:924] Time spent T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.238s user 0.001s sys 0.000s
I20260806 04:01:44.039316 4956 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:44.077425 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 160695us
I20260806 04:01:44.085427 4816 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.087585 4460 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.088527 4784 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:44.040307 4638 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:44.084331 32477 raft_consensus.cc:1261] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:44.092196 4949 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.093962 4892 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:44.095250 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.096526 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:44.098141 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.104148 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33964 (request call id 812) took 1034 ms. Trace:
I20260806 04:01:44.307183 32474 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:44.166246 1137 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:44.292376 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 192643us
I20260806 04:01:44.125779 1188 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 10: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:44.306766 32738 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:44.308245 32469 raft_consensus.cc:2379] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:44.114621 32603 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 04:01:44.282711 32740 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:44.066756 4929 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.128728 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 797) took 1375 ms. Trace:
I20260806 04:01:44.309118 32737 rpcz_store.cc:276] 0806 04:01:43.064552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:43.069287 (+ 4735us) service_pool.cc:227] Handling call
0806 04:01:44.097785 (+1028498us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400245c54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d25cf2a 00007fd172199c6e 00007fd16b890807 00007fd16b890792 00007fd16b8906c1 00007fd16b89063c 00007fd16b88f8f2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:44.098426 (+ 641us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":304747392}
I20260806 04:01:44.137017 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:44.284641 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.307884 32553 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:44.344696 32741 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:44.346305 32553 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:44.347556 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:44.123042 32321 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39754 (request call id 341) took 16563 ms (client timeout 15000 ms). Trace:
I20260806 04:01:44.130786 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:44.066183 4922 raft_consensus.cc:686] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:44.354898 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.349735 1187 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:44.345089 1190 rpcz_store.cc:276] 0806 04:01:42.753568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:42.777482 (+ 23914us) service_pool.cc:227] Handling call
0806 04:01:42.777747 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:42.842252 (+ 64505us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 04:01:42.848168 (+ 5916us) log.cc:842] Serialized 14 byte log entry
0806 04:01:43.501852 (+653684us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b18002590c0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb57b 00007fd16d25e88a
0806 04:01:43.567802 (+ 65950us) spinlock_profiling.cc:243] Waited 40.2 ms on lock 0x7b18002590c0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a32c764 00007fd16a32fb11 00007fd16a32fac9 00007fd16a32fa59 00007fd16a32fa21 00007fd16a32ed1d 00007fd173590204 00007fd173590039 00007fd16a30edc6
0806 04:01:43.679184 (+111382us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:43.679239 (+ 55us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:43.679279 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:44.128466 (+449187us) spinlock_profiling.cc:243] Waited 64.3 ms on lock 0x7b54004e0f54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:44.128521 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:44.128663 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1131,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":3476,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":4254,"spinlock_wait_cycles":248420608,"thread_start_us":169689,"threads_started":1,"wal-append.queue_time_us":1030779}
I20260806 04:01:44.353644 32682 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:44.132928 4943 log.cc:924] Time spent T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.604s user 0.001s sys 0.002s
I20260806 04:01:44.136399 4783 consensus_queue.cc:237] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.122659 4542 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.241s
I20260806 04:01:44.132892 4007 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:44.289934 4486 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.356191 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.363684 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.109611 4884 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:44.069576 4647 consensus_queue.cc:1046] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.280s
I20260806 04:01:44.285017 4264 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.364615 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.362072 32427 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:44.371433 32610 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:44.368619 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:44.389925 1187 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:44.390614 32466 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:44.130615 32324 catalog_manager.cc:5495] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 6 to 8, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 8 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:44.394953 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.396780 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.372318 1189 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:44.374048 1192 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:44.312196 4950 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.312548 4091 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.377221 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:44.377964 32741 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:44.401321 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:44.358103 32682 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:44.386741 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:44.350080 4929 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:44.389442 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.115525 4936 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.395663 32603 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:44.080297 4855 consensus_queue.cc:237] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.355196 4915 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:44.355379 4939 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:44.413547 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.412484 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.413604 32469 raft_consensus.cc:2379] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:44.413195 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:44.411755 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 49764us
W20260806 04:01:44.412050 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 54002.4us
I20260806 04:01:44.393388 32324 catalog_manager.cc:5495] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 2 to 5, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 5 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:44.364208 4836 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:44.429145 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.368836 4912 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.420568 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:44.420202 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.434579 32602 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:44.360961 4182 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.426481 32324 catalog_manager.cc:5495] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 4 to 6, leader changed from d4a238f055a44f459c62c224daa4d541 (127.29.237.132) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 6 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:44.427343 32682 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:44.428717 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.361330 4955 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:44.369338 4921 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.355567 4892 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:44.371425 4816 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:44.429804 1190 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:44.357645 4488 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:44.281766 4806 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.411494 4789 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.091s
I20260806 04:01:44.413632 4264 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.387434 4679 raft_consensus.cc:686] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:44.441524 32324 catalog_manager.cc:5495] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:44.415742 4091 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:44.534934 4091 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:44.536895 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.538273 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.539086 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:44.540868 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:44.442796 4912 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.542990 32469 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260806 04:01:44.484627 4922 consensus_queue.cc:237] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:44.544783 4912 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.545363 4912 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:44.545436 32324 catalog_manager.cc:5495] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 3 to 5, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 5 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:44.546537 32608 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:44.494871 32741 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:44.547035 32324 catalog_manager.cc:5495] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 2 to 4, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 4 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:44.415817 4460 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:44.492213 4182 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.467546 32682 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:44.477979 4439 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:44.500682 32602 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:44.536805 4956 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:44.409703 1187 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 10.
I20260806 04:01:44.362457 4891 consensus_queue.cc:1046] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20260806 04:01:44.424930 4915 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.424011 4656 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 04:01:44.501936 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.551748 32324 catalog_manager.cc:5495] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 4 to 5, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 5 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:44.511015 4892 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.553486 4892 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:44.554064 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:44.554713 32741 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:44.504789 4803 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:44.374488 4745 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:44.348315 32321 rpcz_store.cc:269] 0806 04:01:27.559490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:27.559962 (+ 472us) service_pool.cc:227] Handling call
0806 04:01:44.122947 (+16562985us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:40.688391 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:40.690716 (+ 2325us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:40.690769 (+ 53us) write_op.cc:435] Acquired schema lock
0806 04:01:40.690802 (+ 33us) tablet.cc:660] Decoding operations
0806 04:01:40.725893 (+ 35091us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:40.726057 (+ 164us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:40.726084 (+ 27us) tablet.cc:683] Acquiring locks for 8 operations
0806 04:01:40.726961 (+ 877us) tablet.cc:699] Row locks acquired
0806 04:01:40.727000 (+ 39us) write_op.cc:260] PREPARE: finished
0806 04:01:40.727177 (+ 177us) write_op.cc:270] Start()
0806 04:01:40.727345 (+ 168us) write_op.cc:276] Timestamp: P: 1785988900727105 usec, L: 0
0806 04:01:40.727388 (+ 43us) op_driver.cc:331] REPLICATION: starting
0806 04:01:40.749248 (+ 21860us) log.cc:842] Serialized 3632 byte log entry
0806 04:01:41.368794 (+619546us) spinlock_profiling.cc:243] Waited 8.2 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:41.761582 (+392788us) op_driver.cc:444] REPLICATION: finished
0806 04:01:41.994696 (+233114us) write_op.cc:301] APPLY: starting
0806 04:01:42.016130 (+ 21434us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:42.016371 (+ 241us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:42.016387 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:42.017531 (+ 1144us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:42.017851 (+ 320us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0806 04:01:42.017895 (+ 44us) write_op.cc:312] APPLY: finished
0806 04:01:42.018231 (+ 336us) log.cc:842] Serialized 65 byte log entry
0806 04:01:42.018409 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:42.018531 (+ 122us) write_op.cc:454] Released schema lock
0806 04:01:42.018597 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:42.018921 (+ 324us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:42.018947 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19984,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":187656,"apply.run_cpu_time_us":3453,"apply.run_wall_time_us":77255,"num_ops":8,"prepare.queue_time_us":128837,"prepare.run_cpu_time_us":24211,"prepare.run_wall_time_us":279177,"raft.queue_time_us":111352,"raft.run_cpu_time_us":22314,"raft.run_wall_time_us":105132,"replication_time_us":1034085,"spinlock_wait_cycles":18852096,"thread_start_us":388380,"threads_started":4,"wal-append.queue_time_us":307595,"wal-append.run_cpu_time_us":47512,"wal-append.run_wall_time_us":2035797}]]}
W20260806 04:01:44.510522 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 344364us
I20260806 04:01:44.581100 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:44.569602 32737 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 04:01:44.582074 1137 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:44.418052 4950 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.585023 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.587076 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.432789 4902 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.289009 4914 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.385633 4486 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.425920 1192 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 04:01:44.424597 4396 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260806 04:01:44.498600 4886 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:44.438318 4659 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:44.549021 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 108146us
I20260806 04:01:44.503576 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.553355 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.600179 32470 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:44.600593 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:44.601065 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:44.601555 1137 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:44.578987 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:44.602216 32467 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:44.602490 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:44.585734 32466 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:44.605197 32682 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:44.549084 4264 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:44.601751 32740 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 10: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:44.585721 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:44.602130 32745 raft_consensus.cc:1261] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 04:01:44.590474 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 59985.1us
I20260806 04:01:44.607744 4816 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:44.620944 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:44.621685 32749 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 3.
I20260806 04:01:44.550890 4182 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:44.344964 4949 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.608647 32615 raft_consensus.cc:2354] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:44.638033 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.638684 32470 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:44.598449 4439 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:44.613322 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.598160 4886 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.626005 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:44.557150 4915 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:44.583978 32324 catalog_manager.cc:5495] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: config changed from index 3 to 4, term changed from 4 to 6, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to <none>, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129), VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130), VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131), VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:44.648033 32740 raft_consensus.cc:2379] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:44.597720 4784 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:44.632022 32473 raft_consensus.cc:1226] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:44.605713 4830 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:44.605657 1137 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:44.652531 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 46951us
I20260806 04:01:44.645567 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:44.514611 4921 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:44.648937 32324 catalog_manager.cc:5495] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 2 to 4, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 4 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:44.559358 4979 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:44.635491 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.657913 32602 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:44.651656 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.613955 32604 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:44.628299 4912 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:44.544615 4679 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.653760 1137 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:44.662441 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:44.586357 4007 consensus_queue.cc:1046] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260806 04:01:44.586501 4488 consensus_queue.cc:1046] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
W20260806 04:01:44.663359 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 235740us
I20260806 04:01:44.665241 32324 catalog_manager.cc:5495] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:44.643581 4439 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.603536 4902 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.645058 4659 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:44.634817 4671 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 04:01:44.631560 4571 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:44.697187 4921 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:44.677362 1189 raft_consensus.cc:3032] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:44.697867 1189 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:44.698644 1189 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.699769 1189 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:44.683353 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.700415 32614 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:44.620643 4091 consensus_queue.cc:1035] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20260806 04:01:44.700600 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:44.701299 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:44.701867 32614 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:01:44.702585 1188 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 935) took 1797 ms (client timeout 2142 ms). Trace:
W20260806 04:01:44.702972 1188 rpcz_store.cc:269] 0806 04:01:42.905315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:42.905463 (+ 148us) service_pool.cc:227] Handling call
0806 04:01:44.702408 (+1796945us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54004efcd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:44.702528 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2681475968}
I20260806 04:01:44.683282 32324 catalog_manager.cc:5495] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 3 to 5, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129). New cstate: current_term: 5 leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:44.650292 4886 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:44.686107 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:44.632593 4264 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:44.681667 1137 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:44.694082 4789 raft_consensus.cc:3032] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 04:01:44.695674 4981 log.cc:924] Time spent T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260806 04:01:44.663483 4912 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.675582 4266 raft_consensus.cc:2781] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:44.707790 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.706723 32324 catalog_manager.cc:5495] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 4 to 5, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 5 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:01:44.663797 4860 consensus_queue.cc:1035] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260806 04:01:44.679991 4830 raft_consensus.cc:686] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:44.729000 32467 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:44.689862 4439 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:44.655364 4460 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260806 04:01:44.729875 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:44.730181 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:44.768575 32602 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:44.701017 4091 consensus_queue.cc:1225] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer dd7344d85a154539b7f8c484f2a76772 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 04:01:44.718564 32614 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:44.764055 1189 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:44.662981 4784 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.771459 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:44.781718 32477 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:44.783179 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:44.772418 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:44.784590 32470 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:44.772933 32683 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:44.786370 32683 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:44.705585 4914 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.683116 4803 consensus_queue.cc:1046] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260806 04:01:44.642580 4980 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
I20260806 04:01:44.774569 32324 catalog_manager.cc:5495] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 2 to 3, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } }
I20260806 04:01:44.786880 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:44.705888 4991 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:44.723507 4789 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:44.722139 4659 raft_consensus.cc:686] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:44.732462 4912 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:44.819638 32477 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:44.800155 32602 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:44.722860 4007 consensus_queue.cc:1035] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
W20260806 04:01:44.780892 4849 consensus_meta.cc:287] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.744s user 0.009s sys 0.001s
I20260806 04:01:44.804771 32614 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 9. Current term is 10. Ops: []
I20260806 04:01:44.822202 32553 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:44.802903 32324 catalog_manager.cc:5495] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 2 to 4, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 4 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:44.809747 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:44.781890 4806 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:44.790877 4571 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:44.730433 4264 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:44.808552 4812 consensus_meta.cc:287] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.745s user 0.004s sys 0.005s
I20260806 04:01:44.826439 32614 raft_consensus.cc:1261] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 04:01:44.827400 32324 catalog_manager.cc:5495] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 reported cstate change: term changed from 3 to 5, leader changed from f7c202b562ce482db283463d7f02e136 (127.29.237.131) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 5 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260806 04:01:44.829007 32602 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:44.830145 32682 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:44.803973 4991 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.810642 4956 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:44.850023 32602 raft_consensus.cc:1226] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:44.821866 4922 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:44.803051 4996 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:44.825646 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 910) took 1256 ms. Trace:
I20260806 04:01:44.847795 4571 raft_consensus.cc:686] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:44.859408 4991 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:44.814932 4745 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.826051 4656 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.823160 4486 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:44.831055 4921 raft_consensus.cc:2781] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:44.829598 4914 consensus_queue.cc:1046] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
W20260806 04:01:44.859421 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 137954us
I20260806 04:01:44.826036 4849 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.866863 4979 raft_consensus.cc:686] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:44.870761 4854 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:44.873051 4091 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election won for term 8
W20260806 04:01:44.909046 32739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2382) took 1450 ms (client timeout 1709 ms). Trace:
I20260806 04:01:44.870601 4922 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.875066 4996 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:44.848953 4812 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.865270 4947 consensus_queue.cc:1035] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260806 04:01:44.859473 4830 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:44.919739 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 59450.2us
I20260806 04:01:44.921293 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:44.870256 4400 consensus_queue.cc:1046] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
W20260806 04:01:44.929425 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 99336.8us
I20260806 04:01:44.870366 4789 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.913153 4816 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:44.867599 4891 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260806 04:01:44.914785 4266 raft_consensus.cc:686] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:44.935439 1185 raft_consensus.cc:2354] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8.
W20260806 04:01:44.929418 32739 rpcz_store.cc:269] 0806 04:01:43.440324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:43.440464 (+ 140us) service_pool.cc:227] Handling call
0806 04:01:44.890364 (+1449900us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54000393d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:44.890494 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":986142080}
W20260806 04:01:44.977423 4644 consensus_meta.cc:287] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.061s user 0.011s sys 0.001s
I20260806 04:01:44.932739 4854 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:44.979327 32682 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:44.929425 4914 consensus_queue.cc:1225] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer dd7344d85a154539b7f8c484f2a76772 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260806 04:01:44.928695 4921 raft_consensus.cc:686] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:44.990698 32612 raft_consensus.cc:1261] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260806 04:01:44.933796 4922 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:44.994757 32612 pending_rounds.cc:76] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 3
I20260806 04:01:44.998797 32612 pending_rounds.cc:97] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 04:01:44.917014 4656 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:44.999150 32612 raft_consensus.cc:2866] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:44.933993 4996 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:45.005306 32682 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:44.783998 4856 raft_consensus.cc:2932] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:44.936118 4091 raft_consensus.cc:686] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:44.981343 4816 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:44.908500 4659 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.033314 32602 pending_rounds.cc:76] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 2
I20260806 04:01:45.038637 32602 pending_rounds.cc:97] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 04:01:45.038998 32602 raft_consensus.cc:2866] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:44.998190 4784 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.036870 5008 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:45.040299 4816 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:45.048425 5002 log.cc:924] Time spent T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20260806 04:01:45.040576 5009 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:45.074321 5009 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.075774 4812 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.046751 5027 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:44.875566 1182 rpcz_store.cc:276] 0806 04:01:43.567455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.120227 (+552772us) service_pool.cc:227] Handling call
0806 04:01:44.120584 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:44.120828 (+ 244us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:44.121196 (+ 368us) log.cc:842] Serialized 14 byte log entry
0806 04:01:44.781777 (+660581us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:44.784192 (+ 2415us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:44.794977 (+ 10785us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:44.807635 (+ 12658us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:44.824220 (+ 16585us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4510,"prepare.run_cpu_time_us":7067,"prepare.run_wall_time_us":180934,"raft.queue_time_us":63584,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":88,"thread_start_us":572430,"threads_started":2,"wal-append.queue_time_us":476749}
W20260806 04:01:45.079417 4929 consensus_meta.cc:287] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.649s user 0.007s sys 0.003s
I20260806 04:01:45.079860 4929 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:45.052857 32427 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 2.142s (SENT) [suppressed 8 similar messages]
I20260806 04:01:45.066170 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2158) took 1542 ms. Trace:
I20260806 04:01:45.080976 32612 rpcz_store.cc:276] 0806 04:01:43.523974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:43.524288 (+ 314us) service_pool.cc:227] Handling call
0806 04:01:43.524584 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:44.999631 (+1475047us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:45.065951 (+ 66320us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540034f3d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:45.066115 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3370809216}
I20260806 04:01:45.085304 4950 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.086416 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:45.086876 4637 raft_consensus.cc:3032] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:45.087270 4637 raft_consensus.cc:729] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:45.090422 5009 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.092308 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:45.092803 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:45.093477 32608 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:45.095405 4955 raft_consensus.cc:3032] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:45.095803 4955 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:44.987128 4571 consensus_queue.cc:237] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.071900 4921 consensus_queue.cc:237] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.054383 4914 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:45.073078 4854 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.037701 4979 consensus_queue.cc:237] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.095221 4891 consensus_queue.cc:1035] T 18f3a6d6cef24470a69dbafa39b3d3e1 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:45.096071 32427 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:45.097433 32477 raft_consensus.cc:1261] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:01:45.056320 5008 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:45.071457 5014 log.cc:924] Time spent T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260806 04:01:45.081931 4929 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.099998 32610 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 2168) took 1149 ms. Trace:
I20260806 04:01:45.049861 4264 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.125965 32477 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260806 04:01:45.137769 32610 rpcz_store.cc:276] 0806 04:01:43.950252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:43.950510 (+ 258us) service_pool.cc:227] Handling call
0806 04:01:45.099833 (+1149323us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b5400086954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:45.099941 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":824167168}
I20260806 04:01:45.143949 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2161) took 1605 ms. Trace:
I20260806 04:01:45.144342 32609 rpcz_store.cc:276] 0806 04:01:43.538223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:43.538462 (+ 239us) service_pool.cc:227] Handling call
0806 04:01:43.549875 (+ 11413us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:43.884171 (+334296us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:45.143707 (+1259536us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54000541d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:45.143878 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":511630208}
W20260806 04:01:45.150952 4999 log.cc:924] Time spent T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.221s user 0.000s sys 0.001s
W20260806 04:01:45.140152 4997 log.cc:924] Time spent T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.254s user 0.000s sys 0.001s
I20260806 04:01:45.136180 5008 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:45.141695 4860 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:45.162567 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:45.167780 32604 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:45.168987 4679 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:45.169888 4679 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.155225 4854 raft_consensus.cc:1050] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: attempting to promote NON_VOTER d4a238f055a44f459c62c224daa4d541 to VOTER
W20260806 04:01:45.175552 5013 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20260806 04:01:45.177520 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:45.205116 32477 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:45.148586 4950 consensus_queue.cc:1046] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260806 04:01:45.149915 4849 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:45.206317 32683 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:45.111343 4914 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.196727 32610 raft_consensus.cc:1261] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:01:45.207691 32604 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:45.208493 4914 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.061111 4266 consensus_queue.cc:237] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.218116 1183 raft_consensus.cc:1261] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:01:45.139585 4976 consensus_queue.cc:1035] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260806 04:01:45.215155 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:45.102228 4637 consensus_queue.cc:260] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.215819 5009 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:01:45.209264 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 298033us
I20260806 04:01:45.233340 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:45.234148 32682 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:45.234771 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:45.234702 32427 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:45.235600 32604 raft_consensus.cc:2379] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:45.240038 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:45.173013 4836 consensus_meta.cc:287] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.731s user 0.008s sys 0.002s
I20260806 04:01:45.096659 4955 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.167622 4860 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:45.221612 4915 consensus_meta.cc:287] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.546s user 0.011s sys 0.000s
I20260806 04:01:45.247891 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:45.136392 4091 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.249400 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:45.182298 4929 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.212934 4656 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:45.229619 4884 consensus_meta.cc:287] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.831s user 0.011s sys 0.000s
I20260806 04:01:45.269203 1186 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:45.269727 4481 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.270190 4481 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.231793 5009 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:45.263170 4915 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.263402 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:45.280723 1188 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 10: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:45.264196 32605 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:45.248780 4972 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.251644 4486 raft_consensus.cc:2781] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:45.283977 4955 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:45.284667 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:45.284816 4679 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.232863 5026 consensus_queue.cc:1035] T bf5ed335f5bb4cf491d1d169c809b5d9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20260806 04:01:45.286398 4787 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.254755 4860 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:45.283455 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 49404.8us
I20260806 04:01:45.298723 32610 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260806 04:01:45.285936 1188 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260806 04:01:45.325409 4745 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:45.326694 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:45.279837 5009 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:45.327307 32604 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:45.326156 4486 raft_consensus.cc:686] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:45.336603 32427 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:45.372892 32428 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:45.326027 4745 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.374879 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:45.318172 5057 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.278586 4929 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:45.391618 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 55103.3us
I20260806 04:01:45.326476 1183 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:45.328701 5059 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:01:45.374878 4892 consensus_meta.cc:287] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.821s user 0.012s sys 0.000s
W20260806 04:01:45.388731 4912 consensus_meta.cc:287] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.558s user 0.005s sys 0.005s
I20260806 04:01:45.328217 4787 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.330085 5054 consensus_queue.cc:1046] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260806 04:01:45.369578 4679 consensus_queue.cc:1046] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260806 04:01:45.397460 5059 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:45.410151 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:45.420185 1188 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:45.396450 4914 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 04:01:45.436522 5050 log.cc:924] Time spent T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
W20260806 04:01:45.436668 5034 log.cc:924] Time spent T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260806 04:01:45.301189 4972 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.442458 32741 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:45.397014 5061 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:45.400851 4892 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.423834 4914 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:45.463351 4914 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:45.411831 4912 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.451290 5061 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:45.464771 5066 log.cc:924] Time spent T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 04:01:45.464531 4836 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.464558 5061 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:45.467101 4836 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.466780 1187 raft_consensus.cc:2354] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8.
W20260806 04:01:45.470407 4991 consensus_meta.cc:287] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.556s user 0.009s sys 0.000s
I20260806 04:01:45.427781 5059 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:45.496908 4991 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.497700 1188 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:45.499333 1188 raft_consensus.cc:1261] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:45.499961 4581 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:45.500730 1188 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 04:01:45.500643 4581 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.502862 4581 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.503451 4679 consensus_queue.cc:1225] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer dd7344d85a154539b7f8c484f2a76772 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260806 04:01:45.504420 5049 raft_consensus.cc:3032] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 04:01:45.504820 5049 raft_consensus.cc:729] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:45.505668 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:45.505481 5049 consensus_queue.cc:260] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.506261 32469 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:45.499346 4991 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.396126 5057 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.521534 32469 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:45.529152 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:45.495507 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:45.511256 5026 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.518257 4892 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.536016 4879 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:45.552567 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:45.454950 4486 consensus_queue.cc:237] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.559369 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:45.552423 1184 raft_consensus.cc:2354] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5.
I20260806 04:01:45.568771 1188 raft_consensus.cc:2354] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5.
I20260806 04:01:45.567198 4836 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:45.553841 4922 consensus_meta.cc:287] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.527s user 0.008s sys 0.003s
I20260806 04:01:45.550493 4571 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.549970 4915 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.601373 4922 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.602463 32740 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 8: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:45.603343 4922 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.604423 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 925) took 1239 ms. Trace:
I20260806 04:01:45.604770 1185 rpcz_store.cc:276] 0806 04:01:44.364709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.412936 (+ 48227us) service_pool.cc:227] Handling call
0806 04:01:45.604237 (+1191301us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b5400710554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:45.604364 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1171981696}
I20260806 04:01:45.560742 4091 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:45.591615 5085 log.cc:924] Time spent T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260806 04:01:45.610854 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:45.591981 4836 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.591434 5005 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:45.615620 1188 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 10: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
W20260806 04:01:45.608803 5027 consensus_meta.cc:287] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.532s user 0.013s sys 0.000s
I20260806 04:01:45.616542 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:45.590269 4745 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.600358 4929 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.619326 32477 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:01:45.623468 5086 log.cc:924] Time spent T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20260806 04:01:45.622413 5065 log.cc:924] Time spent T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260806 04:01:45.619441 4856 consensus_meta.cc:287] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.555s user 0.010s sys 0.000s
I20260806 04:01:45.639705 32477 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:45.546360 4854 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:45.630985 4886 consensus_meta.cc:287] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.865s user 0.012s sys 0.000s
I20260806 04:01:45.546002 5026 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.605794 5059 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.652288 32683 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:45.625921 5027 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:45.624265 4915 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.621649 5058 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:45.610920 4571 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.615355 4091 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.658578 32477 raft_consensus.cc:1261] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:01:45.659515 4656 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:45.661830 4571 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:45.637216 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:45.663249 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:45.665100 4439 consensus_meta.cc:287] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.884s user 0.009s sys 0.003s
I20260806 04:01:45.621160 5005 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.621215 4836 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:45.667400 5084 log.cc:924] Time spent T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20260806 04:01:45.686239 4816 consensus_meta.cc:287] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.585s user 0.009s sys 0.000s
I20260806 04:01:45.662191 5058 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.669366 5057 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.670629 4902 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.689085 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 619) took 1093 ms. Trace:
W20260806 04:01:45.687786 5092 log.cc:924] Time spent T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260806 04:01:45.669476 4922 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.663553 4972 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:45.698658 4996 consensus_meta.cc:287] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.660s user 0.002s sys 0.008s
I20260806 04:01:45.687786 5059 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.714253 4836 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:45.714340 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:45.714957 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 218994us
I20260806 04:01:45.716421 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:45.656208 4886 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.697989 4784 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:45.679016 4860 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:45.716343 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:45.721163 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:45.675459 4915 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.723337 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:45.725940 32748 rpcz_store.cc:276] 0806 04:01:44.595993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.596150 (+ 157us) service_pool.cc:227] Handling call
0806 04:01:44.628217 (+ 32067us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:44.900946 (+272729us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:01:45.044673 (+143727us) log.cc:842] Serialized 14 byte log entry
0806 04:01:45.222583 (+177910us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:45.248166 (+ 25583us) log.cc:842] Serialized 52 byte log entry
0806 04:01:45.248408 (+ 242us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:45.248439 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:45.688681 (+440242us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b54003dfed4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:45.688730 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:45.688775 (+ 45us) raft_consensus.cc:1671] finished
0806 04:01:45.688882 (+ 107us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:45.689025 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6666,"prepare.queue_time_us":223025,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":70170,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":16914,"spinlock_wait_cycles":8867200,"thread_start_us":463601,"threads_started":3,"wal-append.queue_time_us":212593}
I20260806 04:01:45.717721 4656 raft_consensus.cc:686] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:45.717919 4439 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:45.718214 5027 consensus_queue.cc:1035] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
W20260806 04:01:45.729753 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 131090us
I20260806 04:01:45.722891 4816 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.120198 4656 consensus_queue.cc:237] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.121636 4816 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:45.803790 4396 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:46.122400 5009 consensus_meta.cc:287] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.754s user 0.006s sys 0.002s
W20260806 04:01:46.124387 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.20306e+06us
I20260806 04:01:46.124483 1136 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:46.124908 32608 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:46.127146 4996 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.128573 1182 raft_consensus.cc:2354] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9.
I20260806 04:01:46.122925 5009 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:46.123499 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 364432us
W20260806 04:01:46.029704 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 228284us
I20260806 04:01:45.734941 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:45.790330 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 68675.1us
I20260806 04:01:46.124176 32615 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:46.141220 1137 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:45.846776 4955 consensus_meta.cc:287] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.547s user 0.004s sys 0.006s
W20260806 04:01:45.848914 5008 consensus_meta.cc:287] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.673s user 0.008s sys 0.003s
W20260806 04:01:46.093842 5113 log.cc:924] Time spent T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260806 04:01:46.145570 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.146426 1136 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:46.161229 32614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1983) took 1355 ms. Trace:
W20260806 04:01:45.827389 5065 log.cc:924] Time spent T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260806 04:01:46.161551 4797 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 04:01:46.161826 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.162245 4797 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:46.162369 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 1984) took 1129 ms. Trace:
I20260806 04:01:46.150316 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.151189 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.162729 32470 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:46.149772 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.164949 32475 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:46.152225 5047 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:46.144073 4858 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.009s
I20260806 04:01:46.153172 4886 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.153327 4439 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.153928 32477 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:45.845772 5026 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.154520 4488 raft_consensus.cc:3032] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:46.155505 1189 raft_consensus.cc:2396] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260806 04:01:46.143929 5059 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260806 04:01:46.157338 5008 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:45.875433 5077 log.cc:924] Time spent T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.231s user 0.000s sys 0.000s
I20260806 04:01:45.837268 4884 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.139639 4264 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:45.877326 4956 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.200881 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:45.840173 5102 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:45.762229 4806 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 04:01:45.920361 4746 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.170449 1136 leader_election.cc:398] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:46.208758 1136 leader_election.cc:401] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:46.171037 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.150816 32466 raft_consensus.cc:2932] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:46.161633 32614 rpcz_store.cc:276] 0806 04:01:44.806184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.839155 (+ 32971us) service_pool.cc:227] Handling call
0806 04:01:45.100483 (+261328us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:45.110247 (+ 9764us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:45.110272 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:45.110707 (+ 435us) log.cc:842] Serialized 26 byte log entry
0806 04:01:45.246240 (+135533us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:45.333889 (+ 87649us) log.cc:842] Serialized 14 byte log entry
0806 04:01:45.334127 (+ 238us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:45.334206 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:45.334244 (+ 38us) raft_consensus.cc:1671] finished
0806 04:01:45.334255 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:46.161084 (+826829us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b5400086950. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:46.161180 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":75211,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":13558,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":46810,"spinlock_wait_cycles":600201600,"thread_start_us":162741,"threads_started":2,"wal-append.queue_time_us":156500}
I20260806 04:01:46.162755 32602 rpcz_store.cc:276] 0806 04:01:45.032515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.032711 (+ 196us) service_pool.cc:227] Handling call
0806 04:01:45.033101 (+ 390us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:45.050773 (+ 17672us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:45.050807 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:45.051381 (+ 574us) log.cc:842] Serialized 52 byte log entry
0806 04:01:45.369641 (+318260us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:45.399830 (+ 30189us) log.cc:842] Serialized 14 byte log entry
0806 04:01:45.402797 (+ 2967us) log.cc:842] Serialized 14 byte log entry
0806 04:01:45.403418 (+ 621us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:45.403503 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:45.686505 (+283002us) spinlock_profiling.cc:243] Waited 4.85 ms on lock 0x7b1800387700. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a32c638 00007fd16a32fb11 00007fd16a32fac9 00007fd16a32fa59 00007fd16a32fa21 00007fd16a32ed1d 00007fd173590204 00007fd173590039 00007fd16a30edc6
0806 04:01:46.162183 (+475678us) raft_consensus.cc:1671] finished
0806 04:01:46.162202 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:46.162310 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2562,"prepare.queue_time_us":77082,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":3871,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":17674,"spinlock_wait_cycles":11157888,"thread_start_us":271954,"threads_started":1,"wal-append.queue_time_us":417345}
I20260806 04:01:46.173713 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:46.176614 4849 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:46.177603 4949 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.179615 32605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42618 (request call id 712) took 1812 ms. Trace:
I20260806 04:01:46.180712 5027 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.183137 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.169296 4991 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.133249 4784 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.192644 4488 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:46.130960 4091 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:46.187179 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:46.146900 5094 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.193567 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:46.214145 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.214496 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.163093 4797 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.163627 5058 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.215459 1185 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:45.748407 4783 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.437s
I20260806 04:01:46.198776 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:46.221132 32737 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:46.199064 32748 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:46.173653 5067 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:46.201512 32749 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 5
I20260806 04:01:46.165062 4996 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:46.139555 4998 consensus_queue.cc:1035] T 57c0e905630e43668379f2f95684775f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260806 04:01:46.196468 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.225979 32477 raft_consensus.cc:3037] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:46.147064 4922 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.234335 32746 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:46.236877 4922 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.205955 5009 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.238054 32737 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:46.209484 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 837) took 2405 ms. Trace:
I20260806 04:01:46.239282 32749 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:46.238572 32471 rpcz_store.cc:276] 0806 04:01:43.797550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:43.814334 (+ 16784us) service_pool.cc:227] Handling call
0806 04:01:43.880937 (+ 66603us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:43.910246 (+ 29309us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:43.910275 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:43.913980 (+ 3705us) log.cc:842] Serialized 26 byte log entry
0806 04:01:44.705587 (+791607us) spinlock_profiling.cc:243] Waited 7.04 ms on lock 0x7b180024f700. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb411 00007fd16d2285cf
0806 04:01:44.843093 (+137506us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:44.926461 (+ 83368us) log.cc:842] Serialized 14 byte log entry
0806 04:01:44.933765 (+ 7304us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:46.202976 (+1269211us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b540009db54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:46.203029 (+ 53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:46.203075 (+ 46us) raft_consensus.cc:1671] finished
0806 04:01:46.203086 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:46.203202 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10189,"prepare.queue_time_us":91028,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":12027,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":6208,"spinlock_wait_cycles":82166144,"thread_start_us":196872,"threads_started":2,"wal-append.queue_time_us":984000}
I20260806 04:01:46.212708 1190 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 10.
I20260806 04:01:46.211726 32605 rpcz_store.cc:276] 0806 04:01:44.367395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.367761 (+ 366us) service_pool.cc:227] Handling call
0806 04:01:46.179456 (+1811695us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b5400305a54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:46.179556 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1993023744}
I20260806 04:01:46.215341 32478 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:46.240676 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.219796 5027 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.243635 5027 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.215004 32738 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:46.212175 4849 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:46.224571 5094 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.257596 32738 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
W20260806 04:01:46.215277 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 44966us
I20260806 04:01:46.223785 32470 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:46.224965 32475 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:46.228439 4488 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.230489 5008 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:46.140303 4396 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.212455 32745 raft_consensus.cc:2354] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7.
I20260806 04:01:45.876046 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:46.241238 32471 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:46.197420 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:45.892510 4914 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.219894 4991 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.247145 32478 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:46.247547 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:46.265223 1137 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:46.251668 32737 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:46.258800 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.265544 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.204342 4884 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.220185 4856 raft_consensus.cc:3037] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 6
I20260806 04:01:46.199034 5119 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:46.260056 32551 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:46.266060 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.266214 32475 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:46.266160 32470 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 8: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:46.253360 4849 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:46.272570 32477 raft_consensus.cc:2454] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:46.273394 1185 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 8: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:46.252193 5067 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.275426 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.214583 4746 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.279175 32470 raft_consensus.cc:1226] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:46.251575 4797 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:46.271169 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.273969 1137 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:46.276013 1185 raft_consensus.cc:2362] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5. Re-sending same reply.
I20260806 04:01:46.205265 4956 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.267760 32749 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:46.278129 1188 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:46.277633 32551 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:46.270272 4488 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:46.293469 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.296151 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.303224 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:46.294281 1137 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:46.304683 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:46.305778 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.275439 4991 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.320904 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.322881 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.324247 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.324898 1185 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:46.282961 4998 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:46.274528 4915 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.325915 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:46.315966 32475 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:46.273164 5119 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.328665 1137 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:46.212778 5102 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.282177 4784 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:46.282742 1183 raft_consensus.cc:3037] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:46.310156 4783 raft_consensus.cc:2781] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:46.352617 4783 raft_consensus.cc:686] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:46.353636 4783 consensus_queue.cc:237] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.285211 4860 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.356312 4998 raft_consensus.cc:686] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:46.327976 5061 consensus_meta.cc:287] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.817s user 0.006s sys 0.006s
I20260806 04:01:46.357254 4998 consensus_queue.cc:237] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.357597 5061 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.359010 32469 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:46.320282 4644 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:46.357264 4396 leader_election.cc:288] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.362483 4884 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.365829 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 938) took 1154 ms. Trace:
I20260806 04:01:46.366488 1182 rpcz_store.cc:276] 0806 04:01:45.211703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.214817 (+ 3114us) service_pool.cc:227] Handling call
0806 04:01:46.365473 (+1150656us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54006b9ed4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:46.365706 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2058280192}
I20260806 04:01:46.342401 5119 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.343572 5094 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.343973 4806 raft_consensus.cc:2781] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:46.372958 4806 raft_consensus.cc:686] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:46.375511 1186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 921) took 2011 ms (client timeout 1813 ms). Trace:
I20260806 04:01:46.374759 4806 consensus_queue.cc:237] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:46.345232 4879 consensus_meta.cc:287] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.786s user 0.006s sys 0.004s
I20260806 04:01:46.347008 4746 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.348696 5067 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.348989 4836 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.350028 4956 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.336403 4991 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:46.350555 5102 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:46.375946 1186 rpcz_store.cc:269] 0806 04:01:44.363604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.364323 (+ 719us) service_pool.cc:227] Handling call
0806 04:01:46.375327 (+2011004us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b5400397e54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:46.375439 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2024145024}
I20260806 04:01:46.382921 4264 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:01:46.387118 4523 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Advancing to term 10
W20260806 04:01:46.436710 4854 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.001s sys 0.027s
I20260806 04:01:46.472733 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:46.474123 4460 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:46.475175 4460 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:46.475660 4460 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:46.392395 1188 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:46.393175 32471 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 9.
I20260806 04:01:46.394259 4912 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.413955 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2176) took 1330 ms. Trace:
I20260806 04:01:46.416704 5061 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.419420 4849 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.415951 32603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:41296 (request call id 1109) took 1917 ms. Trace:
I20260806 04:01:46.424782 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:46.487841 32612 rpcz_store.cc:276] 0806 04:01:45.083454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.083668 (+ 214us) service_pool.cc:227] Handling call
0806 04:01:45.083946 (+ 278us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:45.084301 (+ 355us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:45.084337 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:45.085939 (+ 1602us) log.cc:842] Serialized 26 byte log entry
0806 04:01:46.413260 (+1327321us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b540026aaa0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d1e4028 00007fd16d254a7b 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c
0806 04:01:46.413331 (+ 71us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:46.413386 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:46.413470 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:46.413521 (+ 51us) raft_consensus.cc:1671] finished
0806 04:01:46.413533 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:46.413871 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1223,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":7822592,"thread_start_us":53805,"threads_started":1,"wal-append.queue_time_us":48037,"wal-append.run_cpu_time_us":2003,"wal-append.run_wall_time_us":1021631}
W20260806 04:01:46.328540 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 183465us
W20260806 04:01:46.435168 1184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 804) took 1780 ms (client timeout 1690 ms). Trace:
I20260806 04:01:46.425869 32749 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
W20260806 04:01:46.451974 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 313305us
I20260806 04:01:46.452203 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2409) took 1218 ms. Trace:
I20260806 04:01:46.391475 1183 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:46.452523 4679 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:46.504606 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 185221us
I20260806 04:01:46.506244 32551 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:46.507336 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:46.464538 4787 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.508769 4912 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:46.509189 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:46.510528 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:46.511099 32738 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 7
I20260806 04:01:46.513397 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:46.516520 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:46.517230 32739 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:46.469691 4637 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:46.470295 4991 raft_consensus.cc:686] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:46.519026 4991 consensus_queue.cc:237] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.439064 4972 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:46.345074 4914 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.458843 4836 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.524883 4836 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.524950 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.477466 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:46.472383 4854 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s user 0.001s sys 0.027s
I20260806 04:01:46.479756 5123 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.482170 4894 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:46.482352 5128 consensus_queue.cc:1035] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260806 04:01:46.526654 5123 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.526800 4894 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.527112 4679 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.528394 32466 raft_consensus.cc:1261] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260806 04:01:46.528723 32466 pending_rounds.cc:76] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 4
I20260806 04:01:46.528926 32466 pending_rounds.cc:97] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260806 04:01:46.529151 4894 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.484547 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:46.494721 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 70009.5us
I20260806 04:01:46.525852 1190 raft_consensus.cc:2362] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 10. Re-sending same reply.
I20260806 04:01:46.496183 32682 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:46.495944 32743 rpcz_store.cc:276] 0806 04:01:45.233316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.236160 (+ 2844us) service_pool.cc:227] Handling call
0806 04:01:45.253796 (+ 17636us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:45.407793 (+153997us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:45.407867 (+ 74us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:45.485355 (+ 77488us) log.cc:842] Serialized 26 byte log entry
0806 04:01:45.606845 (+121490us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:01:45.606885 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:46.438128 (+831243us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54000f9e54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:46.438199 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:46.438251 (+ 52us) raft_consensus.cc:1671] finished
0806 04:01:46.438265 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:46.452129 (+ 13864us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1639,"raft.queue_time_us":2104,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":15961,"spinlock_wait_cycles":349366656,"thread_start_us":67320,"threads_started":1,"wal-append.queue_time_us":144347}
I20260806 04:01:46.498689 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:46.508152 32737 raft_consensus.cc:2379] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:46.530512 32470 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:46.529165 32466 raft_consensus.cc:2866] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:46.530958 4993 consensus_queue.cc:1046] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:46.526394 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:46.476754 4922 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:46.532675 4922 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } }
I20260806 04:01:46.533509 4993 raft_consensus.cc:3032] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:46.533876 4993 raft_consensus.cc:729] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:46.534786 4922 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:46.529865 1186 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:46.534683 4993 consensus_queue.cc:260] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.530656 1137 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:46.529085 5123 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.536832 32749 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:46.525755 32475 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 7
I20260806 04:01:46.537006 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.535938 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.537650 1190 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:46.537770 32470 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:46.538405 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:46.538550 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:46.539216 1190 raft_consensus.cc:2362] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5. Re-sending same reply.
I20260806 04:01:46.541245 4884 raft_consensus.cc:2781] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:46.541788 4884 raft_consensus.cc:686] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [term 9 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:46.541982 1137 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:46.542932 4847 raft_consensus.cc:2726] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:46.543376 4784 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:46.543399 4884 consensus_queue.cc:237] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.531358 32682 leader_election.cc:398] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8.
I20260806 04:01:46.542339 4996 consensus_queue.cc:1046] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.004s
I20260806 04:01:46.546598 32682 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:46.546710 4996 raft_consensus.cc:3032] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:46.547341 4996 raft_consensus.cc:729] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:46.493808 32603 rpcz_store.cc:276] 0806 04:01:44.498349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.501207 (+ 2858us) service_pool.cc:227] Handling call
0806 04:01:46.415170 (+1913963us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b5400306954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d25cf2a 00007fd172199c6e 00007fd16b890807 00007fd16b890792 00007fd16b8906c1 00007fd16b89063c 00007fd16b88f8f2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:46.415801 (+ 631us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69977472}
I20260806 04:01:46.485659 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.567233 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.565701 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:46.585652 32475 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:46.510550 32551 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:46.569830 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.587694 32746 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:46.547356 4922 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:46.592273 32738 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
W20260806 04:01:46.593245 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 82776.8us
I20260806 04:01:46.595695 32551 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:46.597368 32551 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:46.600752 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.601468 32470 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 8: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:46.493189 4860 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:46.604858 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.607239 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:46.607931 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:46.610356 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 44764.9us
I20260806 04:01:46.615754 32427 leader_election.cc:398] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5.
I20260806 04:01:46.616060 32427 leader_election.cc:401] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 04:01:46.565488 4264 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:46.620311 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:46.566000 5027 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:46.588810 5094 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.632311 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.634188 1190 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 10.
I20260806 04:01:46.634740 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:46.635987 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:46.594127 5008 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:46.595675 4993 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:46.638213 32612 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:46.588958 4953 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260806 04:01:46.603587 4787 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:46.643692 32605 raft_consensus.cc:2379] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:46.655206 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:46.648444 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.565184 5057 consensus_queue.cc:1035] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260806 04:01:46.650199 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:46.661113 32478 raft_consensus.cc:2354] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7.
I20260806 04:01:46.613010 4836 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:46.659624 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:46.659792 32428 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:46.664619 4671 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:46.665210 4671 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:46.501128 1184 rpcz_store.cc:269] 0806 04:01:44.655112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.664472 (+ 9360us) service_pool.cc:227] Handling call
0806 04:01:46.434856 (+1770384us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b54005d1454. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:46.435018 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1862764928}
I20260806 04:01:46.667342 4671 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:46.667768 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.669487 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.671491 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.593310 4847 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260806 04:01:46.679399 5005 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:46.619743 4949 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:46.616680 4855 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:46.658872 4914 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5.
W20260806 04:01:46.751515 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 91988.7us
I20260806 04:01:46.773475 32467 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:46.774650 32478 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:46.756453 5061 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:46.675457 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:46.754951 32605 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:46.776623 1190 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 8: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:46.618335 4912 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:01:46.681144 4836 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:46.585464 4929 consensus_queue.cc:1046] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260806 04:01:46.617939 5102 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:46.644361 4264 raft_consensus.cc:686] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:46.756587 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 140948us
I20260806 04:01:46.759470 32683 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:46.763873 32614 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260806 04:01:46.920619 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 161225us
I20260806 04:01:46.921241 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:46.921976 32614 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:46.766494 32428 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:46.765080 32612 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:46.621201 4860 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.741739 5110 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.749723 4784 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:46.757829 4783 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:46.646932 5094 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.784024 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:46.821749 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.908068 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:46.773072 5133 log.cc:924] Time spent T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.127s user 0.000s sys 0.004s
I20260806 04:01:46.777871 5061 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:46.868633 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:46.955497 32609 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:46.632037 4996 consensus_queue.cc:260] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.954012 32605 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
W20260806 04:01:46.961441 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 195041us
I20260806 04:01:46.965701 32738 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:46.865278 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:46.966928 32604 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:46.768226 4856 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6.
I20260806 04:01:46.767760 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:46.815641 5148 log.cc:924] Time spent T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260806 04:01:46.775535 4947 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.777467 4836 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:46.910106 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1727) took 3010 ms. Trace:
I20260806 04:01:46.978565 4991 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:46.978139 32473 rpcz_store.cc:276] 0806 04:01:43.899269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:43.943298 (+ 44029us) service_pool.cc:227] Handling call
0806 04:01:43.943562 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:44.652007 (+708445us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:46.909836 (+2257829us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b5400159fd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:46.910043 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1582900992}
I20260806 04:01:46.808321 4806 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.091s
I20260806 04:01:46.774914 4855 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.982617 1190 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:46.983182 1190 raft_consensus.cc:595] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:46.983493 4855 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:46.868607 5005 raft_consensus.cc:3032] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:46.984615 5005 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:46.985483 5005 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.986879 5005 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:46.987346 5094 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:46.953639 4266 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:46.988128 4266 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.990093 4266 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:46.919584 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 164575us
I20260806 04:01:46.839655 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:46.895510 4912 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:46.933091 32683 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:46.646727 5027 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.958775 32478 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:01:46.954493 32615 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:46.968524 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:46.924791 4007 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.976096 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:46.895129 5102 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:46.937812 5119 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:46.980182 4949 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:46.985955 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 946) took 1698 ms. Trace:
I20260806 04:01:46.952188 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:46.916777 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:46.956195 4784 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:46.956043 5110 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:47.038671 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 76394us
I20260806 04:01:47.282968 32428 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:47.095258 1185 raft_consensus.cc:2454] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:47.061102 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:47.286590 32612 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:47.286825 1187 rpcz_store.cc:276] 0806 04:01:45.287501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.470202 (+182701us) service_pool.cc:227] Handling call
0806 04:01:46.985289 (+1515087us) spinlock_profiling.cc:243] Waited 4.45 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721aa2b0 00007fd1721957f9 00007fd1721952a2 00007fd16b87b684
0806 04:01:46.985479 (+ 190us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:46.985656 (+ 177us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:46.985679 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:46.985720 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:46.985748 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:46.985797 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:46.985900 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10242432}
I20260806 04:01:47.288385 32608 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:47.125962 32604 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
W20260806 04:01:47.281898 32737 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.012s sys 0.043s
W20260806 04:01:47.291356 32737 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s user 0.012s sys 0.043s
W20260806 04:01:46.980013 4523 consensus_meta.cc:287] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.508s user 0.012s sys 0.000s
I20260806 04:01:46.993304 32475 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:46.958278 5061 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:46.984830 4797 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.693s user 0.005s sys 0.005s
W20260806 04:01:47.311318 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 394631us
I20260806 04:01:47.292757 32428 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:47.305384 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:47.306726 32603 raft_consensus.cc:2379] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 2: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:47.063138 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:47.277233 5170 log.cc:924] Time spent T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260806 04:01:46.958205 4849 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:46.977164 5142 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.219s user 0.001s sys 0.001s
W20260806 04:01:47.277029 4993 consensus_meta.cc:287] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.619s user 0.009s sys 0.001s
I20260806 04:01:46.992569 32470 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:47.317708 32604 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 8
I20260806 04:01:47.310688 32612 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:46.986084 4860 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:46.979003 5152 log.cc:924] Time spent T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20260806 04:01:47.357991 1183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 807) took 1667 ms (client timeout 2149 ms). Trace:
W20260806 04:01:47.358407 1183 rpcz_store.cc:269] 0806 04:01:45.690421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.805356 (+114935us) service_pool.cc:227] Handling call
0806 04:01:47.357828 (+1552472us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b54006bec54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.357933 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36668160}
I20260806 04:01:47.360674 4783 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:47.361021 1186 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:47.361389 4783 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.362020 4784 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:47.362627 4784 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.363080 4784 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.371858 5110 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:47.317942 5094 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:47.320384 5163 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:47.097021 4264 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:47.384493 1189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 941) took 2151 ms (client timeout 1806 ms). Trace:
W20260806 04:01:47.385244 32469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59514 (request call id 2056) took 1723 ms (client timeout 1770 ms). Trace:
W20260806 04:01:47.379387 1188 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 811) took 1667 ms (client timeout 2136 ms). Trace:
W20260806 04:01:47.387573 32469 rpcz_store.cc:269] 0806 04:01:45.662142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.662799 (+ 657us) service_pool.cc:227] Handling call
0806 04:01:47.385073 (+1722274us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b54000488d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.385167 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1493475200}
W20260806 04:01:47.387096 1189 rpcz_store.cc:269] 0806 04:01:45.232920 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.233055 (+ 135us) service_pool.cc:227] Handling call
0806 04:01:47.384225 (+2151170us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54006f0ed4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.384366 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2118524800}
I20260806 04:01:47.291139 4929 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.332739 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:47.343717 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:47.395651 32615 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:47.076224 4855 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.275s
I20260806 04:01:47.345664 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:47.344978 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:47.351584 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:47.396530 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 58780.6us
I20260806 04:01:47.309283 4921 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:46.978981 4460 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.359946 1183 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:47.397780 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 960) took 1129 ms. Trace:
I20260806 04:01:47.320145 32427 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:47.326776 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:47.387856 1188 rpcz_store.cc:269] 0806 04:01:45.712275 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.135544 (+423269us) service_pool.cc:227] Handling call
0806 04:01:47.379229 (+1243685us) spinlock_profiling.cc:243] Waited 66.5 ms on lock 0x7b54006103d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.379330 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":153026304}
I20260806 04:01:47.329568 5005 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:47.396560 32612 raft_consensus.cc:2454] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:47.406322 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:47.406984 32612 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:47.053648 4947 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:47.398098 1183 rpcz_store.cc:276] 0806 04:01:46.268597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.359164 (+1090567us) service_pool.cc:227] Handling call
0806 04:01:47.359472 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:47.397533 (+ 38061us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:47.397730 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:47.385119 1187 raft_consensus.cc:1261] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260806 04:01:47.429853 1187 pending_rounds.cc:76] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Aborting all ops after (but not including) 7
I20260806 04:01:47.430119 1187 pending_rounds.cc:97] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260806 04:01:47.430346 1187 raft_consensus.cc:2866] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:47.398429 32746 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:47.400816 32608 raft_consensus.cc:2379] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:47.431178 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.394430 32604 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:47.353652 1184 raft_consensus.cc:2379] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 5: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:47.431205 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.403124 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:47.411670 5157 raft_consensus.cc:2741] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 04:01:47.407706 5094 raft_consensus.cc:686] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:47.412911 4460 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:47.414410 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:47.465826 32740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2410) took 2212 ms (client timeout 1753 ms). Trace:
W20260806 04:01:47.466264 32740 rpcz_store.cc:269] 0806 04:01:45.233604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.239811 (+ 6207us) service_pool.cc:227] Handling call
0806 04:01:47.445567 (+2205756us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400245254. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.445735 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":831781248}
I20260806 04:01:47.352419 1182 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:47.418233 4996 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:47.488257 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.490015 32746 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:47.405731 4797 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:47.422523 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 927) took 1156 ms. Trace:
I20260806 04:01:47.494042 1185 rpcz_store.cc:276] 0806 04:01:46.265639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.325694 (+ 60055us) service_pool.cc:227] Handling call
0806 04:01:47.422339 (+1096645us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54006abb54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.422464 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1263445760}
I20260806 04:01:47.423130 4789 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:47.423739 1190 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:47.496927 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:47.425652 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:47.431847 32475 raft_consensus.cc:1261] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:47.431969 32478 raft_consensus.cc:2379] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 9: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:47.498303 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.333853 4949 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:47.438655 1137 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:47.443142 32604 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:47.382153 4972 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:47.527241 4972 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:47.532298 5172 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:47.532778 5172 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:47.533186 5172 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:47.317354 4007 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.376046 5027 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:47.487685 32603 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 8: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:47.422214 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.550647 32466 raft_consensus.cc:2396] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:01:47.428653 4414 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.493028 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:47.424979 4523 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.497678 32608 raft_consensus.cc:3037] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:47.572124 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.500622 32428 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:47.573942 32469 raft_consensus.cc:1203] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[3.6-5.7] Dedup: 3.6->[5.7-5.7]
I20260806 04:01:47.503432 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:47.505095 5173 log.cc:924] Time spent T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260806 04:01:47.493425 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 928) took 1225 ms. Trace:
I20260806 04:01:47.453460 4912 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.501837 4797 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:47.406177 4929 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.506453 5094 consensus_queue.cc:237] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:47.454910 4439 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260806 04:01:47.512297 4949 raft_consensus.cc:686] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:47.512467 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:47.607803 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:47.502615 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 64055.8us
I20260806 04:01:47.515482 5135 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.515920 4789 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.516754 5102 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:47.524856 4783 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:47.408735 4812 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.091s
I20260806 04:01:47.402487 4921 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.526782 32604 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:47.536991 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:47.465011 4856 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
W20260806 04:01:47.409585 4787 consensus_meta.cc:287] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.737s user 0.010s sys 0.000s
I20260806 04:01:47.435616 5019 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.611631 5135 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.619843 5019 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.445048 5005 raft_consensus.cc:686] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:47.614566 4789 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.621558 5005 consensus_queue.cc:237] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:47.623034 4789 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:47.544305 4784 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:47.452334 5141 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:47.545974 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59514 (request call id 2061) took 1396 ms. Trace:
I20260806 04:01:47.546617 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.546815 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 967) took 1069 ms. Trace:
I20260806 04:01:47.552568 32615 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:47.552466 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.575193 32475 raft_consensus.cc:2379] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 7: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
W20260806 04:01:47.576273 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 75757.7us
I20260806 04:01:47.624971 1186 rpcz_store.cc:276] 0806 04:01:46.477414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.538153 (+ 60739us) service_pool.cc:227] Handling call
0806 04:01:47.546664 (+1008511us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b5400611554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.546759 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1847815552}
I20260806 04:01:47.584532 5159 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:01:47.563640 5161 log.cc:924] Time spent T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260806 04:01:47.618917 32605 raft_consensus.cc:1226] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: [2.4-5.5]
I20260806 04:01:47.598924 32466 raft_consensus.cc:1261] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:47.603883 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:47.608587 32612 raft_consensus.cc:2379] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 7: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:47.624930 32473 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:47.584697 32469 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260806 04:01:47.624539 32467 rpcz_store.cc:276] 0806 04:01:46.149966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.150091 (+ 125us) service_pool.cc:227] Handling call
0806 04:01:47.545815 (+1395724us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b5400158bd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.545913 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1401486592}
I20260806 04:01:47.640015 32608 raft_consensus.cc:2454] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:47.617436 32614 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 10.
I20260806 04:01:47.618193 32749 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:01:47.617224 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:47.625643 32477 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:47.631397 32604 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260806 04:01:47.643013 32604 pending_rounds.cc:76] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 6
I20260806 04:01:47.643255 32604 pending_rounds.cc:97] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 04:01:47.643481 32604 raft_consensus.cc:2866] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:47.632443 32615 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260806 04:01:47.644116 32615 pending_rounds.cc:76] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 7
I20260806 04:01:47.644338 32615 pending_rounds.cc:97] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260806 04:01:47.644546 32615 raft_consensus.cc:2866] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:47.645951 32612 raft_consensus.cc:1226] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260806 04:01:47.649919 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 728651us
I20260806 04:01:47.633225 32475 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:01:47.634382 32743 raft_consensus.cc:2379] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 4: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:47.617516 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:47.653669 1137 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:47.619416 4787 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:47.614728 5102 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.584270 4414 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.589221 1184 rpcz_store.cc:276] 0806 04:01:46.265891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.807788 (+541897us) service_pool.cc:227] Handling call
0806 04:01:47.490964 (+683176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:47.588730 4523 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:47.759446 32610 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.005s sys 0.019s
W20260806 04:01:47.761747 32610 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.769s user 0.005s sys 0.019s
I20260806 04:01:47.716184 32603 raft_consensus.cc:1261] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:01:47.722878 32615 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 04:01:47.764899 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 114268us
W20260806 04:01:47.772063 32745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33994 (request call id 2414) took 2523 ms (client timeout 2080 ms). Trace:
W20260806 04:01:47.772500 32745 rpcz_store.cc:269] 0806 04:01:45.249011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.249161 (+ 150us) service_pool.cc:227] Handling call
0806 04:01:47.771833 (+2522672us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b54003a4dd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:47.771982 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2199134592}
I20260806 04:01:47.711422 4264 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:47.759248 5119 raft_consensus.cc:686] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:47.779132 32683 leader_election.cc:398] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7.
I20260806 04:01:47.779479 32683 leader_election.cc:401] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:47.829399 32683 leader_election.cc:302] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:47.709070 4996 consensus_queue.cc:1035] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
W20260806 04:01:47.837224 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 316756us
W20260806 04:01:47.840886 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 290835us
I20260806 04:01:47.842854 32551 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:47.843614 32553 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:47.846117 32467 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:47.852481 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:47.754542 4812 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20260806 04:01:47.713824 4266 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.126s
I20260806 04:01:47.827665 4884 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.759799 5102 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:47.868286 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 548227us
I20260806 04:01:47.628479 5141 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:47.766855 4542 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260806 04:01:47.880949 1136 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:47.847703 4783 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.855216 4787 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.870826 32427 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:47.856746 5009 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941) with higher term. Message: Invalid argument: T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7.
I20260806 04:01:47.889650 1136 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:47.900628 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:47.916653 32740 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 04:01:47.917466 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 46711.2us
I20260806 04:01:47.777175 4949 consensus_queue.cc:237] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:47.899427 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:47.915251 1183 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:47.930488 1185 raft_consensus.cc:3037] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:01:47.912822 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 285642us
I20260806 04:01:47.923074 32427 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:47.879244 4847 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.872534 5190 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:01:47.925547 32559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.004s sys 0.013s
W20260806 04:01:47.937997 32559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.928s user 0.005s sys 0.013s
W20260806 04:01:47.941359 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 172390us
I20260806 04:01:47.911638 4784 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:47.868243 4915 consensus_queue.cc:1046] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
W20260806 04:01:47.918789 5061 consensus_meta.cc:287] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.602s user 0.008s sys 0.001s
I20260806 04:01:47.911348 4894 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:47.911590 4830 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.919152 4638 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:47.873848 4264 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:47.925264 4903 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:47.947187 1183 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:47.892904 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:47.926750 5009 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:47.868979 4884 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.962303 32427 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:47.967509 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:47.967957 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 125190us
I20260806 04:01:47.892143 5017 consensus_queue.cc:1035] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260806 04:01:47.944164 4929 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:47.917069 4787 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.034158 1185 raft_consensus.cc:2454] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:47.968716 4400 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260806 04:01:47.950016 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:47.913226 4783 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:48.034616 5177 log.cc:924] Time spent T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260806 04:01:47.948438 4784 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:48.046267 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 84049.6us
I20260806 04:01:47.951841 5159 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 04:01:47.951989 5190 raft_consensus.cc:3037] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:47.951607 4894 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:47.936954 5047 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260806 04:01:47.953035 4797 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:48.048945 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:48.042006 32551 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
W20260806 04:01:48.052577 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 209033us
I20260806 04:01:47.933703 4439 consensus_queue.cc:1046] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 04:01:48.058970 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:48.059293 1192 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.000s sys 0.002s
I20260806 04:01:47.942991 4847 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.934125 5119 consensus_queue.cc:237] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.957219 4972 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.956785 5061 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.966451 4264 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.962344 4830 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:47.970901 5009 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.048681 4915 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:48.360812 5141 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:48.246949 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 936) took 1979 ms. Trace:
I20260806 04:01:48.364741 1186 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:48.129894 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.363174 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2063) took 1876 ms. Trace:
I20260806 04:01:48.227347 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1743) took 3541 ms. Trace:
I20260806 04:01:48.347271 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 725) took 1064 ms. Trace:
W20260806 04:01:48.397984 1192 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s user 0.011s sys 0.012s
W20260806 04:01:48.399000 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 357062us
W20260806 04:01:48.351766 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 145818us
I20260806 04:01:48.397713 1190 rpcz_store.cc:276] 0806 04:01:46.267431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.096182 (+828751us) service_pool.cc:227] Handling call
0806 04:01:47.423363 (+327181us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:47.497271 (+ 73908us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.246756 (+749485us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54006abb50. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:48.246886 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":632215296}
I20260806 04:01:48.250541 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.401340 32553 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:48.397758 1183 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:48.404947 32615 raft_consensus.cc:1261] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 04:01:48.405881 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:48.403872 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:48.401026 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 433510us
I20260806 04:01:48.408584 32682 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:48.401870 32604 rpcz_store.cc:276] 0806 04:01:47.282805 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.616842 (+334037us) service_pool.cc:227] Handling call
0806 04:01:47.617155 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:47.643865 (+ 26710us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.346955 (+703090us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b540035c854. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:48.347193 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31949568}
I20260806 04:01:48.419343 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.420068 5221 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:48.420403 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.421018 32615 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:48.421938 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:48.226451 5182 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.342s user 0.001s sys 0.000s
I20260806 04:01:48.422454 32615 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 3.
I20260806 04:01:48.370347 5043 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:48.423161 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.423280 5043 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.383721 4560 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:48.420105 5017 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260806 04:01:48.424525 4560 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.425467 5043 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.361279 5102 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:48.424270 5220 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
I20260806 04:01:48.251449 5008 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:48.426355 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:48.426805 5008 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:48.427098 4560 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.423997 32612 raft_consensus.cc:2396] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 04:01:48.429494 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.430562 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.043637 5128 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:48.426944 32615 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:48.432307 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.046087 4414 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.433382 32553 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:48.432338 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:48.434464 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:48.431615 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.449558 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:48.450840 32743 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 8: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:48.450344 32612 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:48.401340 32471 rpcz_store.cc:276] 0806 04:01:46.487055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.487215 (+ 160us) service_pool.cc:227] Handling call
0806 04:01:46.487585 (+ 370us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:46.596868 (+109283us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:46.596898 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:46.611769 (+ 14871us) log.cc:842] Serialized 52 byte log entry
0806 04:01:46.900938 (+289169us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:47.428029 (+527091us) log.cc:842] Serialized 14 byte log entry
0806 04:01:47.428299 (+ 270us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.351127 (+922828us) spinlock_profiling.cc:243] Waited 41.7 ms on lock 0x7b54002021d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:48.351186 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:48.351255 (+ 69us) raft_consensus.cc:1671] finished
0806 04:01:48.362939 (+ 11684us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.363116 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96098,"prepare.queue_time_us":160337,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":75382,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":41069,"spinlock_wait_cycles":95834880,"thread_start_us":685425,"threads_started":2,"wal-append.queue_time_us":339306}
I20260806 04:01:48.453197 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.453902 32604 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
W20260806 04:01:48.226955 5198 log.cc:924] Time spent T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.284s user 0.001s sys 0.000s
I20260806 04:01:48.401634 32474 rpcz_store.cc:276] 0806 04:01:44.686385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:44.686539 (+ 154us) service_pool.cc:227] Handling call
0806 04:01:44.690390 (+ 3851us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:44.690730 (+ 340us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:44.690755 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:44.691201 (+ 446us) log.cc:842] Serialized 26 byte log entry
0806 04:01:44.861732 (+170531us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:45.152484 (+290752us) log.cc:842] Serialized 14 byte log entry
0806 04:01:45.314902 (+162418us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:45.314989 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:48.223640 (+2908651us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b1800253c60. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a32c638 00007fd16a32fb11 00007fd16a32fac9 00007fd16a32fa59 00007fd16a32fa21 00007fd16a32ed1d 00007fd173590204 00007fd173590039 00007fd16a30edc6
0806 04:01:48.226990 (+ 3350us) raft_consensus.cc:1671] finished
0806 04:01:48.227104 (+ 114us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.227272 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23593,"prepare.queue_time_us":136155,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":107743,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":12600,"spinlock_wait_cycles":13522176,"thread_start_us":516202,"threads_started":3,"wal-append.queue_time_us":315704}
I20260806 04:01:48.461055 32466 raft_consensus.cc:1226] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:48.123314 5159 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:48.224314 5191 log.cc:924] Time spent T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.276s user 0.001s sys 0.000s
I20260806 04:01:48.471359 5141 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:48.477798 5141 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:48.478158 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.449348 5220 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:48.478631 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 641) took 1492 ms. Trace:
W20260806 04:01:48.166952 5175 log.cc:924] Time spent T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.370s user 0.000s sys 0.001s
I20260806 04:01:48.479470 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.457294 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:48.479773 5141 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:48.190292 5169 log.cc:924] Time spent T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
W20260806 04:01:48.481524 5209 log.cc:924] Time spent T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.290s user 0.000s sys 0.000s
I20260806 04:01:48.482604 1190 raft_consensus.cc:1226] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:48.484694 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.485327 32612 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
W20260806 04:01:48.486042 5226 log.cc:924] Time spent T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260806 04:01:48.486485 32612 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:01:48.487797 32612 raft_consensus.cc:1261] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 04:01:48.489449 5061 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.491102 4264 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:48.431761 4439 consensus_queue.cc:1225] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer dd7344d85a154539b7f8c484f2a76772 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 04:01:48.462035 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.493548 4847 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:48.493407 32604 raft_consensus.cc:1261] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:48.489216 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 643) took 1165 ms. Trace:
I20260806 04:01:48.494181 32748 rpcz_store.cc:276] 0806 04:01:47.323590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.323742 (+ 152us) service_pool.cc:227] Handling call
0806 04:01:47.367490 (+ 43748us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:47.367840 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:47.367866 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:47.368372 (+ 506us) log.cc:842] Serialized 26 byte log entry
0806 04:01:47.651267 (+282895us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:47.651308 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:47.652040 (+ 732us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:48.489024 (+836984us) raft_consensus.cc:1671] finished
0806 04:01:48.489043 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.489160 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5700,"raft.run_cpu_time_us":44,"raft.run_wall_time_us":44,"thread_start_us":280889,"threads_started":1,"wal-append.queue_time_us":469510}
I20260806 04:01:48.495605 4972 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.236258 5205 raft_consensus.cc:3032] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:48.054340 4914 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:48.056192 4523 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.245388 5049 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:48.496682 32743 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
W20260806 04:01:48.227666 5184 log.cc:924] Time spent T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.311s user 0.000s sys 0.000s
I20260806 04:01:48.497486 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.468468 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:48.498523 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.121659 4784 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:48.493553 4830 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:48.483942 1190 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:48.464969 32427 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:48.233280 4921 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.229322 4481 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:48.238488 4783 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:48.196020 5152 log.cc:924] Time spent T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 04:01:48.480620 5141 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:48.502434 5141 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:48.502838 5141 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:48.193641 5202 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:48.205579 5194 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:48.496025 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.496411 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.480754 32553 leader_election.cc:302] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:48.545519 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.499801 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:48.545838 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 65168.1us
I20260806 04:01:48.473246 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:48.472270 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.189855 4903 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:48.409891 4894 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:48.463977 1183 raft_consensus.cc:1261] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:01:48.507206 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.547685 32474 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:48.132480 1184 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
W20260806 04:01:48.253037 1182 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 938) took 1654 ms (client timeout 1764 ms). Trace:
W20260806 04:01:48.548431 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 80054.2us
I20260806 04:01:48.239993 5005 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:48.431388 5221 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:48.478991 32738 rpcz_store.cc:276] 0806 04:01:46.985877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.990356 (+ 4479us) service_pool.cc:227] Handling call
0806 04:01:47.280960 (+290604us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:47.289377 (+ 8417us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:47.289416 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:47.304102 (+ 14686us) log.cc:842] Serialized 26 byte log entry
0806 04:01:47.629352 (+325250us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:47.629402 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:47.629469 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:48.478406 (+848937us) raft_consensus.cc:1671] finished
0806 04:01:48.478425 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.478553 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7981,"raft.queue_time_us":8533,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":18316,"thread_start_us":233910,"threads_started":1,"wal-append.queue_time_us":539067}
I20260806 04:01:48.544818 32748 raft_consensus.cc:2454] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 1.
I20260806 04:01:48.167114 4929 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.415s
I20260806 04:01:48.553207 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.669608 32743 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:48.668244 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 943) took 3322 ms. Trace:
W20260806 04:01:48.671819 5153 log.cc:924] Time spent T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 1.772s user 0.003s sys 0.010s
I20260806 04:01:48.456247 4816 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:48.695559 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.503309 5061 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:48.673700 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.674955 32610 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2009) took 1770 ms. Trace:
I20260806 04:01:48.577374 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.702268 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.702358 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.703491 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.644662 32553 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:48.670666 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:48.644862 32745 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.012s sys 0.034s
W20260806 04:01:48.704288 32745 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.652s user 0.012s sys 0.037s
I20260806 04:01:48.704869 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.705503 32743 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:48.703815 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.706447 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.671721 1192 rpcz_store.cc:276] 0806 04:01:45.251700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:45.251818 (+ 118us) service_pool.cc:227] Handling call
0806 04:01:45.254439 (+ 2621us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:45.728394 (+473955us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:45.728429 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:45.760583 (+ 32154us) log.cc:842] Serialized 26 byte log entry
0806 04:01:47.436310 (+1675727us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b18004243c0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb411 00007fd16d2285cf
0806 04:01:48.402346 (+966036us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:01:48.544796 (+142450us) log.cc:842] Serialized 14 byte log entry
0806 04:01:48.545084 (+ 288us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.545169 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:48.545218 (+ 49us) raft_consensus.cc:1671] finished
0806 04:01:48.573523 (+ 28305us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.573712 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":530731,"prepare.queue_time_us":143445,"raft.queue_time_us":470257,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":45348,"spinlock_wait_cycles":18254464,"thread_start_us":965849,"threads_started":1,"wal-append.queue_time_us":2727482}
I20260806 04:01:48.399855 4915 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:48.543453 32615 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:48.502946 32427 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:48.690021 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:48.705452 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.707610 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.671965 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.697146 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.708940 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.709354 32474 raft_consensus.cc:2379] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:48.709549 32745 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:48.710103 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.681339 32682 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:48.710897 32474 raft_consensus.cc:2396] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 04:01:48.711786 5230 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:01:48.712126 4976 consensus_queue.cc:1035] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:01:48.701455 32610 rpcz_store.cc:276] 0806 04:01:46.905008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.905159 (+ 151us) service_pool.cc:227] Handling call
0806 04:01:46.957510 (+ 52351us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:46.960706 (+ 3196us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:01:47.762116 (+801410us) log.cc:842] Serialized 14 byte log entry
0806 04:01:48.207818 (+445702us) log.cc:842] Serialized 14 byte log entry
0806 04:01:48.674643 (+466825us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:48.674692 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:48.674736 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.674797 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.674901 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10203,"prepare.queue_time_us":99447,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":99,"raft.queue_time_us":82852,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":1670907,"threads_started":3,"wal-append.queue_time_us":732380}
I20260806 04:01:48.713526 32682 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:48.714759 4972 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:48.548624 5216 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:48.503773 5159 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:48.706902 32467 raft_consensus.cc:2396] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 04:01:48.461694 5102 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:48.548877 4993 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:48.548856 4836 consensus_meta.cc:287] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.244s user 0.012s sys 0.000s
I20260806 04:01:48.123687 5180 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.481s
W20260806 04:01:48.725484 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 222621us
I20260806 04:01:48.578650 5017 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:48.735244 32427 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:48.682862 4914 raft_consensus.cc:686] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:48.698724 4816 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:48.734144 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:48.734417 32428 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:48.715583 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 71039.8us
I20260806 04:01:48.807647 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:48.808218 1192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:48.548066 5205 raft_consensus.cc:729] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:48.711977 32474 raft_consensus.cc:1261] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:01:48.696130 4787 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.461019 5128 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.796303 32609 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260806 04:01:48.796985 32682 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:48.797201 32603 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:48.800493 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:48.800694 5214 log.cc:924] Time spent T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.246s user 0.000s sys 0.001s
I20260806 04:01:48.693161 5005 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:48.695520 4884 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:48.730818 5231 log.cc:924] Time spent T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260806 04:01:48.553972 4921 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:48.719630 5216 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:48.810276 32553 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:48.796305 4637 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:01:48.812069 1136 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.006s sys 0.009s
I20260806 04:01:48.813200 32608 raft_consensus.cc:1226] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:48.813894 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.821779 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:48.845296 1136 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.649s user 0.006s sys 0.009s
W20260806 04:01:48.828626 32740 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.006s sys 0.019s
W20260806 04:01:48.846163 32740 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s user 0.006s sys 0.019s
I20260806 04:01:48.832443 32746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2441) took 1196 ms. Trace:
I20260806 04:01:48.756567 5017 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:48.759749 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:48.716176 5061 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:48.589694 4903 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:48.693315 1182 rpcz_store.cc:269] 0806 04:01:46.598783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.683393 (+ 84610us) service_pool.cc:227] Handling call
0806 04:01:48.252832 (+1569439us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b54004eeb54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:48.252966 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":830475648}
W20260806 04:01:48.845753 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 809109us
I20260806 04:01:48.790019 5043 consensus_queue.cc:1035] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260806 04:01:48.554322 5202 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.846553 32746 rpcz_store.cc:276] 0806 04:01:47.635708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.642117 (+ 6409us) service_pool.cc:227] Handling call
0806 04:01:47.714674 (+ 72557us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:47.767321 (+ 52647us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:47.767353 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:47.767734 (+ 381us) log.cc:842] Serialized 26 byte log entry
0806 04:01:48.058473 (+290739us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:48.058510 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.058577 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:48.832189 (+773612us) raft_consensus.cc:1671] finished
0806 04:01:48.832214 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.832376 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64663,"raft.queue_time_us":57960,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":56741,"thread_start_us":204355,"threads_started":1,"wal-append.queue_time_us":769342}
I20260806 04:01:48.580364 5194 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.812634 4816 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:48.741288 4855 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
W20260806 04:01:48.883508 5172 consensus_meta.cc:287] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 1.310s user 0.011s sys 0.000s
W20260806 04:01:48.884500 32558 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.000s sys 0.015s
W20260806 04:01:48.884893 32558 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.000s sys 0.016s
I20260806 04:01:48.886698 4784 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.851730 32471 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:48.851861 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.842061 32553 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:48.839051 4414 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:48.815487 5102 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:48.741539 4993 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.854508 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:48.847944 4787 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:01:48.835001 5005 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:48.755930 4836 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.865396 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.900640 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.901185 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:48.825944 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 941) took 2226 ms. Trace:
I20260806 04:01:48.841059 4921 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:48.876116 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2440) took 1240 ms. Trace:
I20260806 04:01:48.854099 5061 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:48.879287 4915 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:48.880424 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.909474 32608 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:48.881446 5219 log.cc:924] Time spent T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.161s user 0.000s sys 0.001s
I20260806 04:01:48.881774 5194 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:48.882131 4830 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:48.851382 5017 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:48.894001 32748 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:48.845770 5216 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:48.884719 4996 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:48.903079 1189 rpcz_store.cc:276] 0806 04:01:46.599680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.389273 (+789593us) service_pool.cc:227] Handling call
0806 04:01:47.427548 (+ 38275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:47.427770 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 04:01:47.834656 (+406886us) log.cc:842] Serialized 14 byte log entry
0806 04:01:48.822494 (+987838us) log.cc:842] Serialized 14 byte log entry
0806 04:01:48.825570 (+ 3076us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:48.825621 (+ 51us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:48.825671 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.825741 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.825873 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":131850,"prepare.queue_time_us":521176,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":39365,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":29883,"thread_start_us":976937,"threads_started":3,"wal-append.queue_time_us":592938}
I20260806 04:01:48.864719 4976 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 04:01:48.905656 32610 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:48.905815 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:48.901868 32474 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:48.907907 4644 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:48.907092 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.949525 32615 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:48.876933 4523 consensus_queue.cc:1035] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.063s
I20260806 04:01:48.931437 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.952996 32612 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:48.932152 1192 raft_consensus.cc:595] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:48.953969 1192 raft_consensus.cc:603] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 04:01:48.894477 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 52512.5us
I20260806 04:01:48.954738 32553 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:48.955858 32553 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:48.957108 32478 raft_consensus.cc:2454] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 2.
I20260806 04:01:48.898074 4414 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:48.854262 4929 consensus_queue.cc:1035] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260806 04:01:48.940675 32602 raft_consensus.cc:2379] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 8: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:48.899103 4855 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:48.897876 5220 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:48.942005 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:49.042456 32608 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:48.909487 4921 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:48.905419 32737 rpcz_store.cc:276] 0806 04:01:47.635375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.635837 (+ 462us) service_pool.cc:227] Handling call
0806 04:01:47.712573 (+ 76736us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:47.720183 (+ 7610us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:47.720213 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:47.767984 (+ 47771us) log.cc:842] Serialized 572 byte log entry
0806 04:01:47.951061 (+183077us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:47.951094 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:47.951159 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:48.875877 (+924718us) raft_consensus.cc:1671] finished
0806 04:01:48.875900 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:48.876036 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":182807,"threads_started":1,"wal-append.queue_time_us":728100}
I20260806 04:01:48.945096 32474 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260806 04:01:49.048811 32474 pending_rounds.cc:76] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 8
I20260806 04:01:49.049139 32474 pending_rounds.cc:97] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260806 04:01:48.950524 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:48.855840 5244 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:48.892146 5172 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:48.958200 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:48.942785 4976 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:49.049386 32474 raft_consensus.cc:2866] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:48.949275 4644 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:48.949685 4783 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:48.951850 5108 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:01:49.058708 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 100571us
I20260806 04:01:48.960255 4414 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:48.937994 4830 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:49.058048 5211 log.cc:924] Time spent T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.395s user 0.000s sys 0.001s
I20260806 04:01:49.063534 5102 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:49.078662 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:49.068945 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:49.079293 1190 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:49.039565 4915 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:48.957688 4797 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:48.899609 4914 consensus_queue.cc:237] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:49.072011 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 275095us
I20260806 04:01:48.945555 4996 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.074146 1188 raft_consensus.cc:1226] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:49.056769 4912 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:49.082953 1136 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:49.104184 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:49.098767 5216 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.055953 4787 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:49.063891 4976 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:48.909987 5205 consensus_queue.cc:260] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.068043 4644 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:49.066764 5230 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:01:49.115429 32682 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:49.113910 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:49.074889 5008 consensus_meta.cc:287] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.648s user 0.009s sys 0.006s
W20260806 04:01:49.109246 5190 consensus_meta.cc:287] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.653s user 0.010s sys 0.000s
I20260806 04:01:49.073474 5009 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.138430 32682 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:49.079092 5102 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:49.058830 5220 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:49.180969 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 446594us
I20260806 04:01:49.059247 4855 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.110920 5268 raft_consensus.cc:2781] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:01:49.215997 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:49.244829 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 106492us
I20260806 04:01:49.216816 5008 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:49.131807 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:49.247983 32428 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.203099 4922 raft_consensus.cc:2781] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:49.205543 5190 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:49.263409 5273 consensus_queue.cc:1035] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:49.126412 4787 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:49.108661 4797 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:49.244904 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:49.214637 5102 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:49.259943 5009 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:49.212487 5205 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:49.281201 4892 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:49.261539 5247 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260806 04:01:49.140558 4903 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.250784 5268 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:49.286316 32604 raft_consensus.cc:1261] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:01:49.313019 5268 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:49.287762 32739 raft_consensus.cc:2379] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 7: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:49.275097 1182 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 10.
I20260806 04:01:49.239372 4830 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:49.319509 4679 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:49.201581 4784 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:49.292939 5264 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.250s user 0.002s sys 0.000s
I20260806 04:01:49.293785 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:49.251690 5253 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:49.264751 5194 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:49.254766 5275 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:49.279709 4836 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.311640 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2434) took 1963 ms. Trace:
W20260806 04:01:49.253153 5210 log.cc:924] Time spent T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.427s user 0.000s sys 0.001s
I20260806 04:01:49.315990 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:49.287525 5202 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:49.296825 5009 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:49.348593 5009 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:49.306958 5128 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.326864 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:49.332144 32738 raft_consensus.cc:2379] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:49.334029 5265 log.cc:924] Time spent T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
I20260806 04:01:49.336072 5008 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:49.338049 5141 consensus_meta.cc:287] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.835s user 0.009s sys 0.003s
I20260806 04:01:49.315636 5119 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:49.276252 5180 consensus_queue.cc:1046] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260806 04:01:49.354364 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:49.346690 32741 rpcz_store.cc:276] 0806 04:01:47.347771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.358773 (+ 11002us) service_pool.cc:227] Handling call
0806 04:01:47.359101 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:47.359316 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:01:47.712066 (+352750us) log.cc:842] Serialized 14 byte log entry
0806 04:01:49.311106 (+1599040us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b180011a8e0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16cee382c 00007fd16a30e9dc 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb57b
0806 04:01:49.311290 (+ 184us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:49.311337 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:49.311388 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:49.311459 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:49.311579 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2609,"prepare.queue_time_us":565368,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":5031,"raft.queue_time_us":6941,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":9063,"spinlock_wait_cycles":6128384,"thread_start_us":571484,"threads_started":2,"wal-append.queue_time_us":558990}
I20260806 04:01:49.369377 32605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 729) took 2017 ms. Trace:
I20260806 04:01:49.346804 5172 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:49.381920 32605 rpcz_store.cc:276] 0806 04:01:47.351710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.639091 (+287381us) service_pool.cc:227] Handling call
0806 04:01:47.639411 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:47.665539 (+ 26128us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:47.665577 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:47.666091 (+ 514us) log.cc:842] Serialized 26 byte log entry
0806 04:01:47.931883 (+265792us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:47.969412 (+ 37529us) log.cc:842] Serialized 14 byte log entry
0806 04:01:48.044205 (+ 74793us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.044387 (+ 182us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:49.369137 (+1324750us) raft_consensus.cc:1671] finished
0806 04:01:49.369156 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:49.369300 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73929,"prepare.queue_time_us":30066,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":4294,"raft.queue_time_us":365242,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":50963,"thread_start_us":265320,"threads_started":1,"wal-append.queue_time_us":1010544}
I20260806 04:01:49.387499 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:49.388679 32605 raft_consensus.cc:2379] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:49.378878 4855 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.392416 5180 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:49.393114 5180 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:49.394250 5159 consensus_meta.cc:287] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.653s user 0.008s sys 0.004s
I20260806 04:01:49.394682 5159 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.396036 32475 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 11: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:49.398317 5180 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.396970 5220 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:49.401963 4998 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.010s sys 0.021s
W20260806 04:01:49.395640 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 291111us
I20260806 04:01:49.406662 1136 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.357008 4787 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:49.416487 4998 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.576s user 0.011s sys 0.021s
I20260806 04:01:49.373543 4996 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:49.377785 5194 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:49.377681 5128 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:49.477029 32553 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.445245 1136 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.475112 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:49.488696 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:49.489377 32473 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:49.478945 32553 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.503854 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.511196 32746 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 9. Current term is 10. Ops: []
W20260806 04:01:49.480547 4816 consensus_meta.cc:287] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.582s user 0.006s sys 0.006s
I20260806 04:01:49.471158 4830 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:49.528448 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:49.383339 5141 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:49.382843 5119 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:49.477557 5275 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:49.533234 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:49.512825 4996 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:01:49.536813 32471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.017s sys 0.035s
I20260806 04:01:49.343312 5253 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.482604 5220 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:49.522671 4789 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:49.401521 5009 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:49.541494 32471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s user 0.017s sys 0.035s
I20260806 04:01:49.592164 32603 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 8: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:49.526185 5163 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:49.533435 5266 log.cc:924] Time spent T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.239s user 0.002s sys 0.000s
W20260806 04:01:49.605803 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 348546us
I20260806 04:01:49.606703 32682 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:49.606673 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:49.608134 32746 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:49.608232 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2080) took 1092 ms. Trace:
I20260806 04:01:49.482738 5194 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.522409 5202 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260806 04:01:49.609491 1137 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:49.608629 32467 rpcz_store.cc:276] 0806 04:01:48.515646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.730295 (+214649us) service_pool.cc:227] Handling call
0806 04:01:48.804262 (+ 73967us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:49.139499 (+335237us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:49.139532 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:49.139971 (+ 439us) log.cc:842] Serialized 26 byte log entry
0806 04:01:49.512176 (+372205us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:49.512219 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:49.512290 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:49.608005 (+ 95715us) raft_consensus.cc:1671] finished
0806 04:01:49.608028 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:49.608173 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10324,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":4407,"thread_start_us":705129,"threads_started":2,"wal-append.queue_time_us":334911}
I20260806 04:01:49.513609 5128 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.590646 4523 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:49.605664 4996 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260806 04:01:49.623711 32682 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:49.626066 32741 raft_consensus.cc:1226] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: []
W20260806 04:01:49.607714 5260 log.cc:924] Time spent T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.196s user 0.001s sys 0.001s
I20260806 04:01:49.629941 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:49.630672 1190 consensus_meta.cc:287] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.551s user 0.007s sys 0.003s
I20260806 04:01:49.637581 1190 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:49.538727 4830 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.533783 4799 consensus_queue.cc:1046] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.086s
I20260806 04:01:49.539671 5190 consensus_queue.cc:1035] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.080s
I20260806 04:01:49.531749 4816 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:49.622005 5261 log.cc:924] Time spent T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.235s user 0.000s sys 0.001s
I20260806 04:01:49.655004 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:49.662571 32429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.023s sys 0.010s
W20260806 04:01:49.674474 32429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.855s user 0.023s sys 0.010s
I20260806 04:01:49.691900 5290 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.060s
I20260806 04:01:49.695829 1137 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:49.702575 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2186) took 2696 ms. Trace:
I20260806 04:01:49.703013 32609 rpcz_store.cc:276] 0806 04:01:47.006541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.066088 (+ 59547us) service_pool.cc:227] Handling call
0806 04:01:47.286513 (+220425us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:48.823088 (+1536575us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:49.702296 (+879208us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540034f8d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:49.702514 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3348161920}
I20260806 04:01:49.627556 4523 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:49.628839 5008 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:49.607318 5163 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.720695 32682 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:49.725589 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 385236us
W20260806 04:01:49.663836 5258 log.cc:924] Time spent T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20260806 04:01:49.664126 4929 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:49.732990 32748 raft_consensus.cc:1226] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:49.734364 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:49.736433 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:49.741245 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:49.745532 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:49.727845 4523 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:49.756947 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:49.772215 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 51995.1us
I20260806 04:01:49.737174 4799 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:49.773077 32682 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:49.599045 5275 consensus_queue.cc:1035] T bd55a7d68ee84cb7a69c962f984f9372 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.154s
I20260806 04:01:49.737016 5159 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.782881 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:49.738636 4929 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:49.800447 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:49.757521 5172 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:49.803790 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:49.814380 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 302501us
I20260806 04:01:49.815178 1136 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.774513 5313 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 04:01:49.737815 5300 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 04:01:49.837579 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 655) took 1891 ms. Trace:
I20260806 04:01:49.787966 5255 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:49.838793 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:49.842159 32740 rpcz_store.cc:276] 0806 04:01:47.945845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.946072 (+ 227us) service_pool.cc:227] Handling call
0806 04:01:47.946407 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:47.950412 (+ 4005us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:47.950440 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:47.957543 (+ 7103us) log.cc:842] Serialized 26 byte log entry
0806 04:01:48.849069 (+891526us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:48.849112 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:48.849184 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:49.837364 (+988180us) raft_consensus.cc:1671] finished
0806 04:01:49.837386 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:49.837517 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19818,"raft.queue_time_us":4094,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":19536,"thread_start_us":727012,"threads_started":1,"wal-append.queue_time_us":1040445}
I20260806 04:01:49.843375 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:49.844120 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:49.844705 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:49.799758 5236 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:49.802026 4929 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:49.854148 5338 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:49.854785 5319 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 04:01:49.800959 4903 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:49.855271 1136 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
W20260806 04:01:49.855783 5047 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.002s sys 0.011s
W20260806 04:01:49.858637 4921 consensus_meta.cc:287] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.798s user 0.009s sys 0.000s
I20260806 04:01:49.805956 5172 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:49.875422 4922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.002s sys 0.017s
I20260806 04:01:49.786211 4799 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:49.851262 4894 consensus_meta.cc:287] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 1.210s user 0.012s sys 0.002s
I20260806 04:01:49.880861 4903 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:49.847138 5300 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:49.882833 5047 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.588s user 0.003s sys 0.011s
I20260806 04:01:49.882525 5172 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:49.877902 5009 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.836862 5305 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:49.879231 5319 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:49.890697 5253 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.890563 5141 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:49.919028 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 957) took 3651 ms. Trace:
W20260806 04:01:49.890025 4922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.002s sys 0.017s
I20260806 04:01:49.922950 1187 rpcz_store.cc:276] 0806 04:01:46.267902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.303479 (+1035577us) service_pool.cc:227] Handling call
0806 04:01:47.384742 (+ 81263us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:47.430770 (+ 46028us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:49.918833 (+2488063us) spinlock_profiling.cc:243] Waited 79.5 ms on lock 0x7b54006f0ed0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:49.918970 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":182803328}
W20260806 04:01:49.924614 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 388878us
I20260806 04:01:49.931878 4799 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:49.914134 5311 log.cc:924] Time spent T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
W20260806 04:01:49.936676 32682 connection.cc:232] RPC call timeout handler was delayed by -1.5639079939574003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
I20260806 04:01:49.886051 4921 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:49.943178 32682 connection.cc:232] RPC call timeout handler was delayed by -1.5153079938609153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
I20260806 04:01:49.911603 5202 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 04:01:49.920792 5061 consensus_meta.cc:287] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.984s user 0.012s sys 0.000s
I20260806 04:01:49.951344 32553 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.911972 5194 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.960799 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:49.893560 4894 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:49.933439 5017 consensus_meta.cc:287] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.984s user 0.010s sys 0.003s
I20260806 04:01:49.922788 5128 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.922694 4830 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:49.915757 5305 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:49.923218 4816 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:49.919305 5319 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:49.925676 4922 raft_consensus.cc:686] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:49.959003 5202 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:49.995551 32749 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
W20260806 04:01:49.998157 4644 consensus_meta.cc:287] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.857s user 0.007s sys 0.004s
I20260806 04:01:49.999742 5202 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:50.000130 4644 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:49.964313 5255 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:49.963986 4996 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:49.981843 4414 consensus_meta.cc:287] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.902s user 0.012s sys 0.000s
W20260806 04:01:49.965261 4914 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.035s sys 0.071s
I20260806 04:01:49.984180 4912 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:49.979545 4787 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:50.009608 32608 raft_consensus.cc:2379] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
W20260806 04:01:49.998157 5312 log.cc:924] Time spent T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260806 04:01:49.993276 5305 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:50.011111 4914 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s user 0.035s sys 0.071s
I20260806 04:01:49.964210 5061 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:50.001498 4789 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:49.879474 5338 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.040904 1190 raft_consensus.cc:1226] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
W20260806 04:01:50.042688 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 197629us
I20260806 04:01:49.983158 5017 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.050861 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.066054 1187 raft_consensus.cc:1261] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:50.056957 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:50.050634 5302 log.cc:924] Time spent T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.236s user 0.001s sys 0.001s
I20260806 04:01:50.010366 4996 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.051699 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.013214 4414 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:50.075397 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 220197us
I20260806 04:01:50.037065 4903 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:50.056277 4976 consensus_meta.cc:287] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.924s user 0.007s sys 0.009s
I20260806 04:01:50.019748 4912 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.221670 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:50.111260 5294 log.cc:924] Time spent T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
I20260806 04:01:50.050022 5019 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.040467 4789 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.213066 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:49.969002 5300 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.039469 4264 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:50.069608 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 82405.6us
I20260806 04:01:50.215728 5027 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:50.224730 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:50.226707 32738 raft_consensus.cc:1226] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:50.056485 4787 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.218534 5255 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.218657 5067 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:50.239123 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 876) took 2196 ms. Trace:
I20260806 04:01:50.239615 1189 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:50.223114 5331 log.cc:924] Time spent T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.188s user 0.006s sys 0.000s
I20260806 04:01:50.217476 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:50.245527 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2207) took 1348 ms. Trace:
I20260806 04:01:50.247480 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.236047 4789 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.224982 4976 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.250015 4797 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:50.251278 5322 log.cc:924] Time spent T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.153s user 0.000s sys 0.009s
W20260806 04:01:50.215029 32551 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 1.574s (SENT) [suppressed 23 similar messages]
I20260806 04:01:50.058002 4922 consensus_queue.cc:237] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.273705 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 742) took 1854 ms. Trace:
I20260806 04:01:50.274127 32604 rpcz_store.cc:276] 0806 04:01:48.437096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.496062 (+ 58966us) service_pool.cc:227] Handling call
0806 04:01:48.496923 (+ 861us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:49.312620 (+815697us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.273429 (+960809us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b5400218254. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.273634 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1814608640}
I20260806 04:01:50.275262 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.262341 32615 rpcz_store.cc:276] 0806 04:01:48.897455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.050116 (+152661us) service_pool.cc:227] Handling call
0806 04:01:49.050494 (+ 378us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:49.115887 (+ 65393us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:49.115918 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:49.131335 (+ 15417us) log.cc:842] Serialized 26 byte log entry
0806 04:01:49.612719 (+481384us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:49.837356 (+224637us) log.cc:842] Serialized 14 byte log entry
0806 04:01:49.838904 (+ 1548us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.094842 (+255938us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b540035b954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.094894 (+ 52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:50.240162 (+145268us) raft_consensus.cc:1671] finished
0806 04:01:50.245373 (+ 5211us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.245479 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":242439,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":11187,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":19676,"spinlock_wait_cycles":134947712,"thread_start_us":498748,"threads_started":2,"wal-append.queue_time_us":664608}
W20260806 04:01:50.240558 5005 consensus_meta.cc:287] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 1.312s user 0.016s sys 0.001s
W20260806 04:01:50.277082 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 62094.7us
I20260806 04:01:50.229995 4947 leader_election.cc:288] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:50.253116 5268 consensus_meta.cc:287] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.933s user 0.017s sys 0.004s
I20260806 04:01:50.262126 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.257115 32477 rpcz_store.cc:276] 0806 04:01:48.042438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.312394 (+269956us) service_pool.cc:227] Handling call
0806 04:01:48.367571 (+ 55177us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:48.905960 (+538389us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:48.905990 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:48.932594 (+ 26604us) log.cc:842] Serialized 26 byte log entry
0806 04:01:49.374025 (+441431us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:49.374067 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.092626 (+718559us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b540026f554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.213071 (+120445us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:50.238817 (+ 25746us) raft_consensus.cc:1671] finished
0806 04:01:50.238837 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.239037 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":753,"raft.queue_time_us":2830,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":19178,"spinlock_wait_cycles":1217599232,"thread_start_us":412921,"threads_started":1,"wal-append.queue_time_us":662512}
I20260806 04:01:50.237969 4414 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.221108 5344 consensus_queue.cc:1035] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260806 04:01:50.256623 5355 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:50.260330 5180 consensus_queue.cc:1046] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260806 04:01:50.237821 5172 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.260461 5067 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:50.266934 5248 log.cc:924] Time spent T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 1.432s user 0.000s sys 0.007s
I20260806 04:01:50.258581 5255 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:50.254729 5027 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.269837 32740 raft_consensus.cc:2379] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 11: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:50.275897 32604 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:50.259865 5300 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:50.283746 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:50.257440 4787 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.286679 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2081) took 1770 ms. Trace:
I20260806 04:01:50.252117 4264 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.282931 4912 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.287401 4921 leader_election.cc:288] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.279345 4894 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:50.275772 5205 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.971s user 0.010s sys 0.006s
I20260806 04:01:50.312031 4787 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.312414 5027 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:50.291563 32614 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 2188) took 3225 ms (client timeout 2862 ms). Trace:
I20260806 04:01:50.294773 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:50.313247 5304 log.cc:924] Time spent T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.422s user 0.001s sys 0.000s
I20260806 04:01:50.313701 32604 raft_consensus.cc:2379] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 9: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:50.314271 5255 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.314558 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.315208 32604 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:50.313702 32609 raft_consensus.cc:2396] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260806 04:01:50.315794 32614 rpcz_store.cc:269] 0806 04:01:47.062739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.062875 (+ 136us) service_pool.cc:227] Handling call
0806 04:01:50.288259 (+3225384us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b5400054bd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.288362 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":689046400}
I20260806 04:01:50.316023 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:50.317206 32604 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:50.317477 5067 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:50.320555 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2208) took 1264 ms. Trace:
I20260806 04:01:50.320950 1189 raft_consensus.cc:2454] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:50.322022 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.320936 32612 rpcz_store.cc:276] 0806 04:01:49.056648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.057729 (+ 1081us) service_pool.cc:227] Handling call
0806 04:01:49.068516 (+ 10787us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:49.303562 (+235046us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:49.303592 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:49.303627 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:49.303658 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.320344 (+1016686us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540034f3d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.320387 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.320502 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":540047104}
W20260806 04:01:50.319873 5315 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.306s user 0.001s sys 0.009s
I20260806 04:01:50.297336 32615 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:50.298743 32471 rpcz_store.cc:276] 0806 04:01:48.515877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.891600 (+375723us) service_pool.cc:227] Handling call
0806 04:01:48.892058 (+ 458us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:48.892443 (+ 385us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:48.892470 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:48.892994 (+ 524us) log.cc:842] Serialized 52 byte log entry
0806 04:01:49.591609 (+698615us) raft_consensus.cc:1631] Marking committed up to 2
0806 04:01:49.591651 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:49.591722 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:50.286459 (+694737us) raft_consensus.cc:1671] finished
0806 04:01:50.286477 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.286611 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":698036,"threads_started":1,"wal-append.queue_time_us":851583}
I20260806 04:01:50.291311 5355 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.282086 5061 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.292732 4976 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 11 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.293499 4644 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.262058 1182 raft_consensus.cc:2362] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7. Re-sending same reply.
I20260806 04:01:50.323940 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 871) took 2711 ms. Trace:
I20260806 04:01:50.338735 32466 rpcz_store.cc:276] 0806 04:01:47.612487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.639252 (+ 26765us) service_pool.cc:227] Handling call
0806 04:01:47.639535 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:48.471560 (+832025us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.323676 (+1852116us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b540009db54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.323854 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1888894336}
I20260806 04:01:50.315907 4264 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:50.339411 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2087) took 1033 ms. Trace:
I20260806 04:01:50.341284 32471 rpcz_store.cc:276] 0806 04:01:49.306388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.326046 (+1019658us) service_pool.cc:227] Handling call
0806 04:01:50.338910 (+ 12864us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.339088 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.339112 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:50.339150 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:50.339184 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.339229 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.339357 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:50.325433 1189 raft_consensus.cc:2396] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 04:01:50.283824 5268 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:50.342087 32615 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:50.333071 32612 raft_consensus.cc:1261] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 04:01:50.326201 5216 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.346668 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:50.322682 5102 consensus_meta.cc:287] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 1.021s user 0.016s sys 0.002s
I20260806 04:01:50.334672 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1765) took 1610 ms. Trace:
I20260806 04:01:50.282968 5005 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.308836 5338 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.340267 32466 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:50.350503 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2089) took 1006 ms. Trace:
I20260806 04:01:50.348340 32474 rpcz_store.cc:276] 0806 04:01:48.723864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.070676 (+346812us) service_pool.cc:227] Handling call
0806 04:01:49.071044 (+ 368us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:49.078416 (+ 7372us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:49.078449 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:49.078990 (+ 541us) log.cc:842] Serialized 52 byte log entry
0806 04:01:49.587144 (+508154us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:49.587188 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:49.587280 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:50.312350 (+725070us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:50.326308 (+ 13958us) raft_consensus.cc:1671] finished
0806 04:01:50.326326 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.334593 (+ 8267us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":136,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":29635,"spinlock_wait_cycles":5055744,"thread_start_us":384824,"threads_started":1,"wal-append.queue_time_us":643754}
I20260806 04:01:50.350826 32466 rpcz_store.cc:276] 0806 04:01:49.344459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.339600 (+995141us) service_pool.cc:227] Handling call
0806 04:01:50.339887 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.350263 (+ 10376us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.350452 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:50.324177 32604 raft_consensus.cc:2454] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:50.332222 4571 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.343123 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.349401 32603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 2211) took 1054 ms. Trace:
I20260806 04:01:50.350750 1183 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:50.344022 4007 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.354919 32603 rpcz_store.cc:276] 0806 04:01:49.295079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.315675 (+ 20596us) service_pool.cc:227] Handling call
0806 04:01:50.349189 (+1033514us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400217354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.349334 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":562999552}
I20260806 04:01:50.358832 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1749) took 2937 ms. Trace:
I20260806 04:01:50.362373 32745 raft_consensus.cc:1226] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:50.367630 5202 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.352267 5102 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:50.368330 32469 rpcz_store.cc:276] 0806 04:01:47.421424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.426154 (+ 4730us) service_pool.cc:227] Handling call
0806 04:01:47.547440 (+121286us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:47.640222 (+ 92782us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.358645 (+2718423us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54001e05d0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.358769 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":278005632}
I20260806 04:01:50.370952 1190 raft_consensus.cc:1226] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 04:01:50.353904 4571 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.372121 5102 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.359617 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:50.367913 32602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2000) took 3713 ms (client timeout 1852 ms). Trace:
I20260806 04:01:50.374620 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:50.374553 32602 rpcz_store.cc:269] 0806 04:01:46.654812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:46.654939 (+ 127us) service_pool.cc:227] Handling call
0806 04:01:50.367711 (+3712772us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b54003a1bd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.367842 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5246427008}
I20260806 04:01:50.376410 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:50.376747 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2084) took 1506 ms. Trace:
I20260806 04:01:50.377030 1189 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:50.377266 32467 rpcz_store.cc:276] 0806 04:01:48.870468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.845680 (+975212us) service_pool.cc:227] Handling call
0806 04:01:49.845992 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:49.915941 (+ 69949us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:49.915974 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:49.916015 (+ 41us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:49.916047 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.376499 (+460452us) spinlock_profiling.cc:243] Waited 69.8 ms on lock 0x7b540019df54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.376545 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.376687 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":160426240}
I20260806 04:01:50.379972 32603 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:50.380683 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:50.383164 32612 raft_consensus.cc:2354] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:50.388797 5119 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:50.370996 5268 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:50.355746 4523 consensus_meta.cc:287] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.577s user 0.008s sys 0.006s
I20260806 04:01:50.359061 4996 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:50.374740 4571 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.364226 5005 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.380336 1182 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:50.393751 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.381812 5202 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.358759 5017 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.398949 5172 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.406847 1189 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:50.398288 5355 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.392800 4523 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:50.408545 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:50.410418 1189 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:50.378367 5305 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.405891 5319 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.414654 5135 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:50.412431 4929 consensus_meta.cc:287] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.527s user 0.007s sys 0.010s
I20260806 04:01:50.424347 4929 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.418207 4858 raft_consensus.cc:3032] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:50.420354 5319 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.425336 4858 raft_consensus.cc:729] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:50.426636 4929 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.426210 4858 consensus_queue.cc:260] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.427377 4858 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:50.430864 5305 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.506708 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.509267 1185 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:50.509506 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.511129 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.512553 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2083) took 1662 ms. Trace:
I20260806 04:01:50.513034 32473 rpcz_store.cc:276] 0806 04:01:48.849517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.522243 (+672726us) service_pool.cc:227] Handling call
0806 04:01:49.525798 (+ 3555us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:49.526002 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 04:01:49.745124 (+219122us) log.cc:842] Serialized 14 byte log entry
0806 04:01:50.478761 (+733637us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b1800267a60. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a2a34ca 00007fd173984f99 00007fd16a167990 00007fd16a329864 00007fd16a329155 00007fd16d24c3d0 00007fd16d25e576 00007fd16d26f0d7 00007fd16d26f058
0806 04:01:50.480221 (+ 1460us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:50.480374 (+ 153us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:50.480529 (+ 155us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.480671 (+ 142us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.512250 (+ 31579us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10273,"prepare.queue_time_us":309155,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"spinlock_wait_cycles":43321088,"thread_start_us":325390,"threads_started":2,"wal-append.queue_time_us":247112}
I20260806 04:01:50.514896 4523 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.410974 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:50.507455 32614 raft_consensus.cc:1226] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:50.506366 32605 raft_consensus.cc:1226] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:50.512010 32612 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:50.526413 4915 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.552523 4523 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.562536 4523 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.564495 4915 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.567647 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.568620 4799 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:50.561106 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.573652 5190 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.574572 5190 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.575765 4523 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.578171 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:50.581602 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.583353 1182 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 10.
I20260806 04:01:50.587015 4929 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.588325 5373 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.589062 4922 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:50.578040 5341 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:50.569473 4799 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.588909 32746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1859) took 1052 ms. Trace:
I20260806 04:01:50.599949 4915 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.602182 32746 rpcz_store.cc:276] 0806 04:01:49.536287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.776152 (+239865us) service_pool.cc:227] Handling call
0806 04:01:50.588374 (+812222us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b5400102a54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.588636 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1795471616}
I20260806 04:01:50.593032 32551 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:50.607515 5373 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.615402 4799 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.616760 5373 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.620217 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.622088 5341 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.623420 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.624320 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:50.629108 4922 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:50.629110 4929 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:50.633539 32749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1850) took 2434 ms (client timeout 1786 ms). Trace:
I20260806 04:01:50.633158 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.635466 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 663) took 1210 ms. Trace:
I20260806 04:01:50.635547 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:50.635936 32738 rpcz_store.cc:276] 0806 04:01:49.425357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.425482 (+ 125us) service_pool.cc:227] Handling call
0806 04:01:49.425811 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.246859 (+821048us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.634972 (+388113us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b54000fb754. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.635291 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1841402240}
I20260806 04:01:50.636474 32610 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 2201) took 1890 ms. Trace:
I20260806 04:01:50.636897 32610 rpcz_store.cc:276] 0806 04:01:48.737812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.739753 (+ 1941us) service_pool.cc:227] Handling call
0806 04:01:50.615996 (+1876243us) spinlock_profiling.cc:243] Waited 60.4 ms on lock 0x7b5400306954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.628452 (+ 12456us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138945920}
I20260806 04:01:50.637871 1189 raft_consensus.cc:2454] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 11.
I20260806 04:01:50.638835 32610 raft_consensus.cc:1203] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[5.6-8.7] Dedup: 5.6->[8.7-8.7]
W20260806 04:01:50.639174 32739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1846) took 2464 ms (client timeout 1596 ms). Trace:
I20260806 04:01:50.639528 32610 raft_consensus.cc:1226] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 8. Current term is 9. Ops: [5.6-8.7]
I20260806 04:01:50.640434 4922 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:50.607717 4523 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:50.636698 5341 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.646431 5190 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.646842 4523 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.639385 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:50.649312 1189 raft_consensus.cc:2354] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 10. Current term is 11.
I20260806 04:01:50.649652 4523 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.652542 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:50.637598 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.642402 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2458) took 1332 ms. Trace:
W20260806 04:01:50.634068 32749 rpcz_store.cc:269] 0806 04:01:48.199345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.471977 (+272632us) service_pool.cc:227] Handling call
0806 04:01:50.633154 (+2161177us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54002120d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.633332 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3330465152}
I20260806 04:01:50.650323 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.635565 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:50.640784 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33182 (request call id 665) took 1169 ms. Trace:
I20260806 04:01:50.656533 32741 rpcz_store.cc:276] 0806 04:01:49.310218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.384381 (+ 74163us) service_pool.cc:227] Handling call
0806 04:01:49.421702 (+ 37321us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:49.628318 (+206616us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.641836 (+1013518us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54000bde54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.642182 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":456911744}
W20260806 04:01:50.644012 32739 rpcz_store.cc:269] 0806 04:01:48.174333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.359832 (+185499us) service_pool.cc:227] Handling call
0806 04:01:50.638627 (+2278795us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b54000393d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.638930 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1947684736}
I20260806 04:01:50.661103 32740 rpcz_store.cc:276] 0806 04:01:49.470857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.843850 (+372993us) service_pool.cc:227] Handling call
0806 04:01:50.640320 (+796470us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b54000f9e54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.640577 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":977048832}
I20260806 04:01:50.665354 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33964 (request call id 815) took 2150 ms. Trace:
I20260806 04:01:50.676424 1189 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:50.676944 32737 raft_consensus.cc:1226] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:50.678691 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2452) took 1465 ms. Trace:
I20260806 04:01:50.679672 1187 raft_consensus.cc:1226] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
W20260806 04:01:50.679473 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 44927.3us
I20260806 04:01:50.687225 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 944) took 2763 ms. Trace:
I20260806 04:01:50.687781 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:50.689393 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:50.687793 1188 rpcz_store.cc:276] 0806 04:01:47.918024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.918164 (+ 140us) service_pool.cc:227] Handling call
0806 04:01:47.918474 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:49.103353 (+1184879us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.680596 (+1577243us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54003faa54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.680945 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2657365888}
I20260806 04:01:50.691195 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.691865 1188 raft_consensus.cc:1226] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
W20260806 04:01:50.692138 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 91584.4us
I20260806 04:01:50.690394 32739 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:50.675765 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.661801 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:50.676972 1186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 813) took 2784 ms (client timeout 1906 ms). Trace:
I20260806 04:01:50.666677 32743 rpcz_store.cc:276] 0806 04:01:48.512310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.712748 (+200438us) service_pool.cc:227] Handling call
0806 04:01:50.622166 (+1909418us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400245c54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d25cf2a 00007fd172199c6e 00007fd16b890807 00007fd16b890792 00007fd16b8906c1 00007fd16b89063c 00007fd16b88f8f2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.662385 (+ 40219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1125488000}
I20260806 04:01:50.679090 32737 rpcz_store.cc:276] 0806 04:01:49.213564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.214973 (+ 1409us) service_pool.cc:227] Handling call
0806 04:01:50.665427 (+1450454us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.677499 (+ 12072us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.678428 (+ 929us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54000393d0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.678620 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3213703552}
I20260806 04:01:50.676357 32612 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 10.
I20260806 04:01:50.692736 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:50.712888 32738 raft_consensus.cc:2396] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260806 04:01:50.690642 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45672 (request call id 10) took 1885 ms. Trace:
I20260806 04:01:50.700134 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 662) took 1727 ms. Trace:
I20260806 04:01:50.697006 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:50.724727 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:50.825989 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:50.829449 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:50.829969 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:50.830358 32474 raft_consensus.cc:2379] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:50.831259 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:50.833600 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:50.667577 4915 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:50.835055 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.835166 4915 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.844916 5163 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:50.837062 5008 raft_consensus.cc:3032] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 04:01:50.813552 1186 rpcz_store.cc:269] 0806 04:01:47.892526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.892648 (+ 122us) service_pool.cc:227] Handling call
0806 04:01:50.676522 (+2783874us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b540013ddd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.676763 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":488726784}
I20260806 04:01:50.831977 32474 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:50.700811 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:50.840945 4644 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:50.817646 1192 rpcz_store.cc:276] 0806 04:01:48.805131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.805253 (+ 122us) service_pool.cc:227] Handling call
0806 04:01:50.690256 (+1885003us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b540044e654. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d24ebc3 00007fd17219881a 00007fd16b88bd04 00007fd16b88bc92 00007fd16b88bbc1 00007fd16b88bb3c 00007fd16b88adf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:50.690426 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":283488640}
I20260806 04:01:50.821792 32748 rpcz_store.cc:276] 0806 04:01:48.904936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.941271 (+ 36335us) service_pool.cc:227] Handling call
0806 04:01:48.941624 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:49.747842 (+806218us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.631451 (+883609us) spinlock_profiling.cc:243] Waited 788 ms on lock 0x7b540011dad4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:50.631895 (+ 444us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1811629312}
I20260806 04:01:50.701757 1187 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:01:50.850793 32741 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:50.847146 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:50.838685 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.692466 5268 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:50.709225 5341 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.163s
I20260806 04:01:50.852569 4644 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.852720 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:50.940397 32682 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:50.912859 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.692814 4523 raft_consensus.cc:2726] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:50.884238 32473 raft_consensus.cc:3037] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:50.943966 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.885915 1188 raft_consensus.cc:1226] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 04:01:50.948514 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:50.882573 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 891) took 1145 ms. Trace:
I20260806 04:01:50.905053 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.912139 32743 raft_consensus.cc:2379] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 9: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:50.882964 5189 raft_consensus.cc:3032] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:50.951061 1184 raft_consensus.cc:2379] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 9: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:50.848340 5008 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:50.940212 1187 raft_consensus.cc:1226] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 04:01:50.877086 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:50.956462 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.952894 5268 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:50.835273 4993 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.940989 4671 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:50.951062 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 127082us
W20260806 04:01:50.975396 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 499531us
I20260806 04:01:50.991405 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:50.992221 32740 raft_consensus.cc:2396] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 04:01:50.994266 32612 raft_consensus.cc:1226] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 04:01:50.893704 4915 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:51.000003 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2086) took 1741 ms. Trace:
I20260806 04:01:51.000633 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:51.005179 5008 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:50.976346 5189 raft_consensus.cc:729] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:50.992611 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:50.996220 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:51.007719 4671 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.008215 5268 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:50.971438 32469 rpcz_store.cc:276] 0806 04:01:49.736865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.372317 (+635452us) service_pool.cc:227] Handling call
0806 04:01:50.372851 (+ 534us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.373318 (+ 467us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 04:01:50.586529 (+213211us) log.cc:842] Serialized 14 byte log entry
0806 04:01:50.874017 (+287488us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:50.881974 (+ 7957us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:50.882100 (+ 126us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.882208 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.882381 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":118835,"prepare.queue_time_us":144203,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":77,"raft.queue_time_us":53560,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":183,"thread_start_us":258689,"threads_started":2,"wal-append.queue_time_us":275526}
I20260806 04:01:51.016541 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:50.997202 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.044975 32737 raft_consensus.cc:2379] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 7: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:50.990655 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:51.028328 32605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2019) took 1211 ms. Trace:
W20260806 04:01:51.028839 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 60821.6us
W20260806 04:01:51.049340 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 129845us
I20260806 04:01:51.049863 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2022) took 1104 ms. Trace:
I20260806 04:01:51.050848 5136 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:51.051638 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2017) took 1250 ms. Trace:
I20260806 04:01:51.003633 1185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:51.039479 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:51.051982 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 112015us
I20260806 04:01:51.062626 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.020911 32477 rpcz_store.cc:276] 0806 04:01:49.250318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.285214 (+1034896us) service_pool.cc:227] Handling call
0806 04:01:50.287478 (+ 2264us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:50.525953 (+238475us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.527746 (+ 1793us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:50.586034 (+ 58288us) log.cc:842] Serialized 26 byte log entry
0806 04:01:50.740157 (+154123us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:50.740198 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.756265 (+ 16067us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:50.975904 (+219639us) raft_consensus.cc:1671] finished
0806 04:01:50.975924 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.991758 (+ 15834us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":191477,"raft.queue_time_us":277386,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":174,"thread_start_us":152692,"threads_started":2,"wal-append.queue_time_us":276830}
I20260806 04:01:51.073324 1185 raft_consensus.cc:595] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:51.073678 5049 consensus_queue.cc:1046] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:51.074947 4679 consensus_queue.cc:1046] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 04:01:51.075418 32682 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:51.076051 4679 raft_consensus.cc:3032] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:51.076543 4679 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:51.078231 5341 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:51.078521 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 657) took 2226 ms. Trace:
I20260806 04:01:51.078863 32745 rpcz_store.cc:276] 0806 04:01:48.851623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.852450 (+ 827us) service_pool.cc:227] Handling call
0806 04:01:48.874119 (+ 21669us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.367711 (+1493592us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.078041 (+710330us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54003a61d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:51.078338 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3422149888}
I20260806 04:01:51.080391 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:51.080545 5110 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:51.081298 5110 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:51.082406 32745 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:51.083626 5110 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:51.084709 5032 raft_consensus.cc:2781] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:51.085358 5032 raft_consensus.cc:686] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:51.086733 5032 consensus_queue.cc:237] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.049758 1182 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:01:51.029031 1184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 948) took 1944 ms (client timeout 2390 ms). Trace:
I20260806 04:01:51.096697 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.075474 1185 raft_consensus.cc:1261] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:51.065160 32608 rpcz_store.cc:276] 0806 04:01:49.801048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.801212 (+ 164us) service_pool.cc:227] Handling call
0806 04:01:51.051216 (+1250004us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54002d0354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:51.051454 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":471236864}
I20260806 04:01:51.090014 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:50.913125 5380 consensus_queue.cc:1035] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.150s
I20260806 04:01:51.090967 1187 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:51.090068 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.059544 32605 rpcz_store.cc:276] 0806 04:01:49.805708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.810552 (+ 4844us) service_pool.cc:227] Handling call
0806 04:01:49.810858 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.528477 (+717619us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.016826 (+488349us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b5400055ad4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:51.017090 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1586816896}
I20260806 04:01:51.077725 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:51.101338 32745 raft_consensus.cc:2454] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 3.
I20260806 04:01:51.098416 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:51.113440 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.122154 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.112982 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:51.147833 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:51.149215 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.149768 32427 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
W20260806 04:01:51.161372 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 83839.3us
I20260806 04:01:51.059960 32604 rpcz_store.cc:276] 0806 04:01:49.838596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.352151 (+513555us) service_pool.cc:227] Handling call
0806 04:01:50.352424 (+ 273us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:50.352756 (+ 332us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.352779 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:50.353205 (+ 426us) log.cc:842] Serialized 26 byte log entry
0806 04:01:50.526797 (+173592us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:50.662503 (+135706us) log.cc:842] Serialized 14 byte log entry
0806 04:01:50.845786 (+183283us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.874476 (+ 28690us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:50.893927 (+ 19451us) raft_consensus.cc:1671] finished
0806 04:01:50.893942 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:50.942551 (+ 48609us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89693,"prepare.queue_time_us":353869,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":130422,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":7906,"thread_start_us":234166,"threads_started":2,"wal-append.queue_time_us":42711}
I20260806 04:01:51.166647 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.173318 32551 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:51.164314 32477 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:51.174805 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.190999 32749 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
W20260806 04:01:51.199611 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 86989.9us
I20260806 04:01:51.193328 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:51.195621 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:51.207350 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 114520us
I20260806 04:01:51.220182 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:51.103646 1184 rpcz_store.cc:269] 0806 04:01:49.073535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:49.079103 (+ 5568us) service_pool.cc:227] Handling call
0806 04:01:50.675389 (+1596286us) spinlock_profiling.cc:243] Waited 9.93 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:50.996714 (+321325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22829952}
I20260806 04:01:51.231307 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.233737 5135 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 04:01:51.234189 5135 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:51.234555 5135 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:51.236336 1187 raft_consensus.cc:1226] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:51.245071 4671 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:51.248047 4993 consensus_queue.cc:1046] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:51.250011 4993 raft_consensus.cc:3032] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:51.250353 4993 raft_consensus.cc:729] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:51.252938 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.200451 5008 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:51.229455 32746 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:51.264354 32739 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:51.266804 4884 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:51.267576 32682 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:51.088155 4679 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:51.274315 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:51.251446 4993 consensus_queue.cc:260] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.291631 32737 raft_consensus.cc:2396] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260806 04:01:51.295075 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:51.292719 32473 raft_consensus.cc:2454] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:51.223132 5049 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:51.300352 4884 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.220947 5341 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:51.301817 5341 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:51.302222 5341 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:51.302991 4884 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:51.310721 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 112287us
I20260806 04:01:51.323156 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.325695 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.327060 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.308945 32467 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 10.
I20260806 04:01:51.322261 5326 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:51.324949 32427 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:51.340577 32741 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:51.341270 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1865) took 1081 ms. Trace:
I20260806 04:01:51.341691 32741 rpcz_store.cc:276] 0806 04:01:50.259674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.692385 (+432711us) service_pool.cc:227] Handling call
0806 04:01:51.341203 (+648818us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:51.211177 5136 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:51.346570 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 70938.9us
I20260806 04:01:51.362991 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.335247 32551 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:51.379895 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:51.361371 32739 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:51.402146 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 253319us
I20260806 04:01:51.392139 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:51.402019 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 171130us
I20260806 04:01:51.406821 32749 raft_consensus.cc:2354] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5.
I20260806 04:01:51.409471 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 669) took 1861 ms. Trace:
I20260806 04:01:51.418924 32738 rpcz_store.cc:276] 0806 04:01:49.529956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.165477 (+1635521us) service_pool.cc:227] Handling call
0806 04:01:51.262117 (+ 96640us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.363853 (+101736us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.363882 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:51.373611 (+ 9729us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:51.391441 (+ 17830us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.391514 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.391646 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:51.391671 4523 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:01:51.483035 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 69622.8us
I20260806 04:01:51.472786 32427 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:51.454154 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:51.484743 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 216334us
I20260806 04:01:51.451574 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.482810 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.452562 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 666) took 1957 ms. Trace:
I20260806 04:01:51.408727 4644 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:51.509281 32740 rpcz_store.cc:276] 0806 04:01:49.476529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.189108 (+1712579us) service_pool.cc:227] Handling call
0806 04:01:51.312107 (+122999us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.409184 (+ 97077us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.409212 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:51.419810 (+ 10598us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:51.434038 (+ 14228us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.434105 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.434217 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:51.310230 5189 consensus_queue.cc:260] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.413690 5110 raft_consensus.cc:2781] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Leader election won for term 11
I20260806 04:01:51.527835 32737 raft_consensus.cc:1261] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:01:51.528654 32737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2461) took 1678 ms. Trace:
I20260806 04:01:51.528991 32737 rpcz_store.cc:276] 0806 04:01:49.850414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.347542 (+1497128us) service_pool.cc:227] Handling call
0806 04:01:51.447093 (+ 99551us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.528324 (+ 81231us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.528583 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:51.451692 5190 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:51.530254 32737 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:51.413436 5163 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:51.367872 5326 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.532482 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:51.532773 5163 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:51.534536 5326 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:51.535312 5163 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:51.508960 5380 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:51.535789 5189 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:51.536322 4644 consensus_queue.cc:1046] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:51.537465 5190 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:51.539686 4523 raft_consensus.cc:3032] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:51.528839 32473 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:51.541040 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:51.541745 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37290 (request call id 982) took 1306 ms. Trace:
I20260806 04:01:51.540146 4523 raft_consensus.cc:729] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:51.543556 4523 consensus_queue.cc:260] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.544154 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:51.545964 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 112725us
I20260806 04:01:51.542068 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:51.556052 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 47179.2us
I20260806 04:01:51.523321 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:51.540123 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 56032.9us
I20260806 04:01:51.446084 4679 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:51.552543 1183 rpcz_store.cc:276] 0806 04:01:50.235549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.261866 (+ 26317us) service_pool.cc:227] Handling call
0806 04:01:51.541402 (+1279536us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b54004853d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:51.541554 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":159667584}
I20260806 04:01:51.541882 1184 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:51.545636 4523 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:51.609640 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.612814 32474 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:51.614459 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:51.617563 5163 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:51.617818 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.619184 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.619880 32737 raft_consensus.cc:2379] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 10: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:51.621763 5135 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.625690 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.627555 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 755) took 1156 ms. Trace:
I20260806 04:01:51.631340 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 835) took 1586 ms. Trace:
I20260806 04:01:51.632679 4993 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:51.631767 1189 rpcz_store.cc:276] 0806 04:01:50.044603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.677397 (+632794us) service_pool.cc:227] Handling call
0806 04:01:50.678012 (+ 615us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:50.678828 (+ 816us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.678862 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:50.680507 (+ 1645us) log.cc:842] Serialized 52 byte log entry
0806 04:01:50.944145 (+263638us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:50.975127 (+ 30982us) log.cc:842] Serialized 14 byte log entry
0806 04:01:51.057746 (+ 82619us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.057822 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:51.631114 (+573292us) raft_consensus.cc:1671] finished
0806 04:01:51.631141 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.631272 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":140843,"prepare.queue_time_us":30632,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":83145,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":333,"thread_start_us":107162,"threads_started":1,"wal-append.queue_time_us":947616}
I20260806 04:01:51.627975 32609 rpcz_store.cc:276] 0806 04:01:50.470938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.658558 (+187620us) service_pool.cc:227] Handling call
0806 04:01:50.659075 (+ 517us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:50.852241 (+193166us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.852280 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:50.852886 (+ 606us) log.cc:842] Serialized 26 byte log entry
0806 04:01:51.154762 (+301876us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:51.218200 (+ 63438us) log.cc:842] Serialized 14 byte log entry
0806 04:01:51.277648 (+ 59448us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.277790 (+ 142us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:51.627322 (+349532us) raft_consensus.cc:1671] finished
0806 04:01:51.627344 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.627495 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":243746,"prepare.queue_time_us":63139,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":229051,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":13802,"thread_start_us":261866,"threads_started":1,"wal-append.queue_time_us":732589}
W20260806 04:01:51.637548 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 64271.6us
I20260806 04:01:51.632462 5163 consensus_queue.cc:1035] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:51.624017 5135 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:51.639662 32473 raft_consensus.cc:1226] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:51.640146 5163 consensus_queue.cc:1046] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:01:51.635617 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 754) took 1165 ms. Trace:
I20260806 04:01:51.640998 4922 consensus_queue.cc:1046] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:01:51.641850 32473 raft_consensus.cc:1226] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 04:01:51.642028 4922 raft_consensus.cc:3032] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:51.642395 4922 raft_consensus.cc:729] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:51.644537 32478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1754) took 4204 ms (client timeout 2008 ms). Trace:
W20260806 04:01:51.644917 32478 rpcz_store.cc:269] 0806 04:01:47.440109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.498052 (+ 57943us) service_pool.cc:227] Handling call
0806 04:01:51.610462 (+4112410us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540009d154. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:51.644320 (+ 33858us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3353360768}
I20260806 04:01:51.640774 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2024) took 1801 ms. Trace:
I20260806 04:01:51.666020 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.667479 4929 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:51.668321 1185 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:51.672242 1187 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
W20260806 04:01:51.673343 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 75595us
I20260806 04:01:51.673623 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:51.674304 32603 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Advancing to term 10
W20260806 04:01:51.676994 32475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1763) took 2984 ms (client timeout 2149 ms). Trace:
W20260806 04:01:51.677378 32475 rpcz_store.cc:269] 0806 04:01:48.692949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:48.703263 (+ 10314us) service_pool.cc:227] Handling call
0806 04:01:51.676628 (+2973365us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b54002cf454. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:51.676809 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1424455552}
I20260806 04:01:51.667496 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.682826 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:51.684636 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 101395us
I20260806 04:01:51.586190 5110 raft_consensus.cc:686] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 11 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:51.686861 32478 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:51.493552 5187 consensus_queue.cc:1035] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.191s
I20260806 04:01:51.690141 32474 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:51.641229 32615 rpcz_store.cc:276] 0806 04:01:50.470561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.659397 (+188836us) service_pool.cc:227] Handling call
0806 04:01:50.696878 (+ 37481us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:50.697450 (+ 572us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.697478 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:50.697987 (+ 509us) log.cc:842] Serialized 26 byte log entry
0806 04:01:50.814567 (+116580us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:50.814604 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.814670 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:51.635419 (+820749us) raft_consensus.cc:1671] finished
0806 04:01:51.635440 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.635564 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1452,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":200,"thread_start_us":113138,"threads_started":1,"wal-append.queue_time_us":934890}
I20260806 04:01:51.694412 32749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2462) took 1842 ms. Trace:
I20260806 04:01:51.694844 32749 rpcz_store.cc:276] 0806 04:01:49.852290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.407744 (+1555454us) service_pool.cc:227] Handling call
0806 04:01:51.595531 (+187787us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.694061 (+ 98530us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.694091 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:51.694146 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:51.694196 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.694254 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.694355 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:51.696569 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.698896 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.643471 4922 consensus_queue.cc:260] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.659472 32602 rpcz_store.cc:276] 0806 04:01:49.839139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.378471 (+539332us) service_pool.cc:227] Handling call
0806 04:01:50.378830 (+ 359us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:50.383744 (+ 4914us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.383775 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:51.075465 (+691690us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b18002f6da0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a2f9f96 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a
0806 04:01:51.076113 (+ 648us) log.cc:842] Serialized 52 byte log entry
0806 04:01:51.344317 (+268204us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:51.402712 (+ 58395us) log.cc:842] Serialized 14 byte log entry
0806 04:01:51.433296 (+ 30584us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.482778 (+ 49482us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:51.640334 (+157556us) raft_consensus.cc:1671] finished
0806 04:01:51.640442 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.640602 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":64449,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":22894,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":84600,"spinlock_wait_cycles":37075584,"thread_start_us":267775,"threads_started":1,"wal-append.queue_time_us":469561}
I20260806 04:01:51.702980 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1782) took 1003 ms. Trace:
I20260806 04:01:51.707151 5049 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:51.707860 5049 raft_consensus.cc:3032] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:51.708226 5049 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:51.710969 32477 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:51.711374 4922 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:51.712950 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 761) took 1240 ms. Trace:
I20260806 04:01:51.713986 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:51.714761 32741 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:51.715361 5110 consensus_queue.cc:237] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.715886 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1870) took 1332 ms. Trace:
I20260806 04:01:51.716526 32745 rpcz_store.cc:276] 0806 04:01:50.383309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.079783 (+696474us) service_pool.cc:227] Handling call
0806 04:01:51.715722 (+635939us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b54002125d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:51.715831 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3216128}
I20260806 04:01:51.717689 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:51.718214 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:51.713330 32608 rpcz_store.cc:276] 0806 04:01:50.472830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.144635 (+671805us) service_pool.cc:227] Handling call
0806 04:01:51.221931 (+ 77296us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:51.251557 (+ 29626us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.251587 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:51.252127 (+ 540us) log.cc:842] Serialized 26 byte log entry
0806 04:01:51.563749 (+311622us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:51.596589 (+ 32840us) log.cc:842] Serialized 14 byte log entry
0806 04:01:51.596817 (+ 228us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.596897 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:51.712710 (+115813us) raft_consensus.cc:1671] finished
0806 04:01:51.712730 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.712884 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25051,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":51,"raft.queue_time_us":8008,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":19602,"thread_start_us":235657,"threads_started":1,"wal-append.queue_time_us":383551}
I20260806 04:01:51.705278 32471 rpcz_store.cc:276] 0806 04:01:50.699386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.568002 (+868616us) service_pool.cc:227] Handling call
0806 04:01:51.684133 (+116131us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.702688 (+ 18555us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.702714 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:51.702752 (+ 38us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:51.702780 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.702831 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.702926 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:51.721758 32749 raft_consensus.cc:2396] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260806 04:01:51.711107 5394 log.cc:924] Time spent T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.092s user 0.001s sys 0.001s
I20260806 04:01:51.731787 32739 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:51.733414 32745 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:51.730468 32603 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 10.
W20260806 04:01:51.735847 32603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2037) took 1268 ms (client timeout 1625 ms). Trace:
I20260806 04:01:51.717586 5190 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:51.736249 32603 rpcz_store.cc:269] 0806 04:01:50.467586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.520547 (+ 52961us) service_pool.cc:227] Handling call
0806 04:01:51.673373 (+1152826us) spinlock_profiling.cc:243] Waited 999 us on lock 0x7b58001e5798. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4bce 000000000034aea9 00007fd172120a30 00007fd1721e67b4 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:51.735732 (+ 62359us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2297600}
I20260806 04:01:51.739463 32610 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 756) took 1268 ms. Trace:
I20260806 04:01:51.740784 32682 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:51.709132 5049 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.742791 5049 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:51.737022 5221 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:51.762588 32553 leader_election.cc:302] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:51.785280 32477 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:51.740739 32610 rpcz_store.cc:276] 0806 04:01:50.471282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.659809 (+188527us) service_pool.cc:227] Handling call
0806 04:01:50.680402 (+ 20593us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:50.836015 (+155613us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.836048 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:50.836762 (+ 714us) log.cc:842] Serialized 26 byte log entry
0806 04:01:50.982404 (+145642us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:50.998650 (+ 16246us) log.cc:842] Serialized 14 byte log entry
0806 04:01:50.999315 (+ 665us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:50.999582 (+ 267us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:51.710309 (+710727us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:51.728960 (+ 18651us) raft_consensus.cc:1671] finished
0806 04:01:51.739258 (+ 10298us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.739402 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155060,"prepare.queue_time_us":17350,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":44,"raft.queue_time_us":182402,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":12287,"spinlock_wait_cycles":23129856,"thread_start_us":134579,"threads_started":1,"wal-append.queue_time_us":781239}
I20260806 04:01:51.792253 4481 consensus_queue.cc:1046] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
W20260806 04:01:51.792671 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 43903.7us
I20260806 04:01:51.793898 32553 leader_election.cc:302] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:51.794493 4875 raft_consensus.cc:2741] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 04:01:51.794989 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:51.795435 4644 raft_consensus.cc:2781] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:51.796033 4915 raft_consensus.cc:2781] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:51.796867 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 971) took 1566 ms. Trace:
I20260806 04:01:51.797266 1190 rpcz_store.cc:276] 0806 04:01:50.230083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.689032 (+458949us) service_pool.cc:227] Handling call
0806 04:01:50.836183 (+147151us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:50.836789 (+ 606us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:50.836824 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:50.837408 (+ 584us) log.cc:842] Serialized 26 byte log entry
0806 04:01:50.988779 (+151371us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:51.022854 (+ 34075us) log.cc:842] Serialized 14 byte log entry
0806 04:01:51.068621 (+ 45767us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.068701 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:51.796457 (+727756us) raft_consensus.cc:1671] finished
0806 04:01:51.796491 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.796649 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45494,"prepare.queue_time_us":33944,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":58213,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":6677,"thread_start_us":134782,"threads_started":1,"wal-append.queue_time_us":951622}
I20260806 04:01:51.748247 5135 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:51.788547 1182 consensus_meta.cc:287] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.632s user 0.005s sys 0.005s
I20260806 04:01:51.803606 1182 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:51.793699 4481 raft_consensus.cc:3032] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 04:01:51.805558 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 47295.7us
I20260806 04:01:51.809428 4875 consensus_queue.cc:1035] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:51.813762 32614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2020) took 2002 ms. Trace:
I20260806 04:01:51.814137 32614 rpcz_store.cc:276] 0806 04:01:49.811437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.317099 (+505662us) service_pool.cc:227] Handling call
0806 04:01:50.317408 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.525071 (+207663us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.813498 (+1288427us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b540034e4d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:51.813704 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":421907840}
I20260806 04:01:51.823177 1192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:51.827082 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 840) took 1582 ms. Trace:
I20260806 04:01:51.828373 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:51.828495 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:51.829783 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 100331us
I20260806 04:01:51.830687 32683 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:51.833293 4875 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:51.833734 4875 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:51.834079 32470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1751) took 4403 ms (client timeout 1596 ms). Trace:
W20260806 04:01:51.834513 32470 rpcz_store.cc:269] 0806 04:01:47.430169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:47.430962 (+ 793us) service_pool.cc:227] Handling call
0806 04:01:51.815934 (+4384972us) spinlock_profiling.cc:243] Waited 2.68 s on lock 0x7b54000488d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:51.833883 (+ 17949us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6158855680}
I20260806 04:01:51.827566 1186 rpcz_store.cc:276] 0806 04:01:50.244336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.878139 (+633803us) service_pool.cc:227] Handling call
0806 04:01:51.547090 (+668951us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.620397 (+ 73307us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.620470 (+ 73us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:51.620543 (+ 73us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:51.620570 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:51.620629 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:51.826877 (+206248us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54004853d0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:51.827017 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1537426816}
I20260806 04:01:51.859439 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:51.823932 1192 raft_consensus.cc:595] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:51.862165 1192 raft_consensus.cc:603] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:51.808528 4481 raft_consensus.cc:729] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:51.868005 4784 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:01:51.869076 4481 consensus_queue.cc:260] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.870260 4481 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:51.858610 5298 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:51.872998 5298 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:51.875177 5298 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:51.875768 5135 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:51.878005 5135 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:51.788247 5136 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:51.881927 1182 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:51.884721 32469 raft_consensus.cc:2379] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:51.834089 4875 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:51.796307 5190 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:01:51.896520 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 56283us
W20260806 04:01:51.902832 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 111927us
W20260806 04:01:51.906411 32746 consensus_meta.cc:287] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.569s user 0.013s sys 0.000s
I20260806 04:01:51.906879 32746 raft_consensus.cc:2454] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:51.907649 32737 raft_consensus.cc:1226] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 04:01:51.885756 4949 raft_consensus.cc:3032] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 04:01:51.915431 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 501657us
I20260806 04:01:51.919698 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:51.926225 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:51.926224 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:51.927275 1182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:51.900640 5135 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:51.928122 1182 raft_consensus.cc:595] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:51.920641 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:51.929585 32741 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:51.927094 1190 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:51.933760 1182 raft_consensus.cc:603] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:51.941699 32683 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:51.938889 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:51.918834 4949 raft_consensus.cc:729] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:51.948187 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:51.936028 1136 connection.cc:232] RPC call timeout handler was delayed by -1.1236200430430472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2066s for a response
I20260806 04:01:51.956585 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
W20260806 04:01:51.961925 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 55682.9us
I20260806 04:01:51.958861 4523 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:51.968065 32427 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:51.968238 5275 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:51.968566 5189 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:51.970167 1192 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:51.940439 4784 consensus_queue.cc:1046] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20260806 04:01:51.963073 5032 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:51.975713 4949 consensus_queue.cc:260] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:51.958194 5135 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:51.976851 5341 consensus_meta.cc:287] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.674s user 0.011s sys 0.004s
I20260806 04:01:51.959412 5136 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:51.977629 1183 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:51.979314 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 04:01:52.004055 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.027992 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:51.980142 1184 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 11.
I20260806 04:01:51.983175 32683 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:51.987859 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:51.999799 4849 consensus_queue.cc:1046] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:51.995455 5298 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:52.002619 32326 catalog_manager.cc:5495] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 5 to 6, leader changed from dd7344d85a154539b7f8c484f2a76772 (127.29.237.130) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 6 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } }
I20260806 04:01:52.014454 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.022966 32553 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:51.987566 5189 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:52.029871 32614 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 11: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:52.030959 4855 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:52.012354 4523 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:52.031877 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 48799.5us
I20260806 04:01:52.031211 5136 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:52.034382 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 66428.9us
I20260806 04:01:52.035333 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 897) took 1024 ms. Trace:
W20260806 04:01:52.009367 5008 consensus_meta.cc:287] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.735s user 0.007s sys 0.003s
I20260806 04:01:52.004482 4784 raft_consensus.cc:3032] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:52.037521 32473 rpcz_store.cc:276] 0806 04:01:51.010473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.642931 (+632458us) service_pool.cc:227] Handling call
0806 04:01:51.643263 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.643605 (+ 342us) raft_consensus.cc:1520] Early marking committed up to index 9
0806 04:01:51.644919 (+ 1314us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.034968 (+390049us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:52.035016 (+ 48us) raft_consensus.cc:1631] Marking committed up to 9
0806 04:01:52.035075 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.035148 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.035273 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":818,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":389548,"threads_started":1}
I20260806 04:01:52.016644 4679 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:52.044234 4637 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:52.044550 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.044883 32427 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:52.046413 32427 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:52.047765 32427 leader_election.cc:398] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Vote denied by peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181) with higher term. Message: Invalid argument: T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 10. Current term is 11.
W20260806 04:01:52.046880 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 58723.7us
I20260806 04:01:52.044924 5298 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.053052 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.038787 4949 raft_consensus.cc:3037] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:52.035938 5189 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:52.036800 5190 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:52.053071 1137 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.055545 32682 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:52.038379 4784 raft_consensus.cc:729] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:52.055253 32603 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:52.056622 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.072808 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:52.074293 32602 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:52.029738 5341 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:52.078635 32400 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet d7c0dfae3c464cd18cfd95401b49eed9
I20260806 04:01:52.079705 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db915af860484020942e9495eda28804" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.077168 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.027719 5032 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.087656 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:52.087754 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:52.089187 32615 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:52.088822 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.090824 4486 raft_consensus.cc:2781] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:52.091233 4486 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:52.091566 4486 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:52.092195 5298 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.081604 4849 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 04:01:52.080385 32400 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet d7c0dfae3c464cd18cfd95401b49eed9
I20260806 04:01:52.090420 32603 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:52.088176 32324 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39766 (request call id 338) took 12137 ms (client timeout 15000 ms). Trace:
I20260806 04:01:52.090547 4789 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:52.095069 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2105) took 1083 ms. Trace:
I20260806 04:01:52.090003 32471 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 10.
I20260806 04:01:52.099507 32738 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:52.088135 32326 catalog_manager.cc:5495] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 2 to 3, leader changed from d4a238f055a44f459c62c224daa4d541 (127.29.237.132) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 3 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:52.075783 5190 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:52.043462 4875 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:52.090790 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:01:52.122179 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.049225 4855 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:52.057621 5008 consensus_queue.cc:1046] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260806 04:01:52.130430 32614 raft_consensus.cc:2379] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:52.099403 5049 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 04:01:52.142380 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.143154 32604 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:52.084441 5380 raft_consensus.cc:686] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:52.144318 1137 leader_election.cc:398] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5.
I20260806 04:01:52.149663 1137 leader_election.cc:401] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 04:01:52.055994 4679 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:52.157090 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 75221.9us
I20260806 04:01:52.132300 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.057621 4523 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.148749 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:52.159217 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:52.151104 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 103525us
I20260806 04:01:52.156399 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:52.108011 5190 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:52.137444 5341 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.100946 32475 rpcz_store.cc:276] 0806 04:01:51.011152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.678199 (+667047us) service_pool.cc:227] Handling call
0806 04:01:51.678571 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.678881 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:01:51.679250 (+ 369us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.089622 (+410372us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.094706 (+ 5084us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:52.094761 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:52.094808 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.094872 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.094988 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15006,"prepare.queue_time_us":1435,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"thread_start_us":392833,"threads_started":1}
W20260806 04:01:52.163528 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 83659.6us
I20260806 04:01:52.160607 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.145637 5049 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:52.171062 4849 raft_consensus.cc:3037] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
W20260806 04:01:52.098526 32324 rpcz_store.cc:269] 0806 04:01:39.950902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:40.181219 (+230317us) service_pool.cc:227] Handling call
0806 04:01:52.081592 (+11900373us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 04:01:45.130111 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 04:01:45.130649 (+ 538us) write_op.cc:432] Acquiring schema lock in shared mode
0806 04:01:45.130688 (+ 39us) write_op.cc:435] Acquired schema lock
0806 04:01:45.130705 (+ 17us) tablet.cc:660] Decoding operations
0806 04:01:45.134522 (+ 3817us) write_op.cc:620] Acquiring the partition lock for write op
0806 04:01:45.134618 (+ 96us) write_op.cc:641] Partition lock acquired for write op
0806 04:01:45.134653 (+ 35us) tablet.cc:683] Acquiring locks for 15 operations
0806 04:01:45.135844 (+ 1191us) tablet.cc:699] Row locks acquired
0806 04:01:45.135867 (+ 23us) write_op.cc:260] PREPARE: finished
0806 04:01:45.143270 (+ 7403us) write_op.cc:270] Start()
0806 04:01:45.152822 (+ 9552us) write_op.cc:276] Timestamp: P: 1785988905135984 usec, L: 0
0806 04:01:45.154416 (+ 1594us) op_driver.cc:331] REPLICATION: starting
0806 04:01:45.248384 (+ 93968us) log.cc:842] Serialized 6009 byte log entry
0806 04:01:45.436930 (+188546us) op_driver.cc:444] REPLICATION: finished
0806 04:01:45.636271 (+199341us) write_op.cc:301] APPLY: starting
0806 04:01:45.645274 (+ 9003us) tablet.cc:1365] starting BulkCheckPresence
0806 04:01:45.760935 (+115661us) tablet.cc:1368] finished BulkCheckPresence
0806 04:01:45.807577 (+ 46642us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 04:01:46.181073 (+373496us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 04:01:46.181331 (+ 258us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0806 04:01:46.181379 (+ 48us) write_op.cc:312] APPLY: finished
0806 04:01:46.181724 (+ 345us) log.cc:842] Serialized 107 byte log entry
0806 04:01:46.181881 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:46.182045 (+ 164us) write_op.cc:454] Released schema lock
0806 04:01:46.182097 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0806 04:01:46.182375 (+ 278us) write_op.cc:489] Releasing partition, row and schema locks
0806 04:01:46.182422 (+ 47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":262259,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":193606,"apply.run_cpu_time_us":6884,"apply.run_wall_time_us":552044,"num_ops":15,"prepare.queue_time_us":154621,"prepare.run_cpu_time_us":10105,"prepare.run_wall_time_us":192888,"raft.queue_time_us":111203,"raft.run_cpu_time_us":18584,"raft.run_wall_time_us":109682,"replication_time_us":282410,"thread_start_us":399982,"threads_started":4,"wal-append.queue_time_us":74480,"wal-append.run_cpu_time_us":25682,"wal-append.run_wall_time_us":1921450}]]}
I20260806 04:01:52.178989 32609 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:52.180442 32604 raft_consensus.cc:2379] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 3: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:52.151119 5422 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:52.180056 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.166337 32326 catalog_manager.cc:5495] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 04:01:52.090749 4784 consensus_queue.cc:260] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.182838 32605 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 10.
I20260806 04:01:52.187211 32427 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:52.189301 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 153480us
I20260806 04:01:52.190343 32682 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:52.191816 5187 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:52.173208 5049 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:52.195541 32553 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.182075 5135 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:52.195731 32326 catalog_manager.cc:5495] T 5c662c1a70954272bb9eff4a51caaeee P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 2 to 4, leader changed from d4a238f055a44f459c62c224daa4d541 (127.29.237.132) to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130). New cstate: current_term: 4 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:52.204177 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:52.199874 5136 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:52.199983 5422 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:52.215710 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:52.221714 32614 raft_consensus.cc:3037] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:52.218674 32326 catalog_manager.cc:5495] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 reported cstate change: term changed from 4 to 6, leader changed from ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129) to f7c202b562ce482db283463d7f02e136 (127.29.237.131). New cstate: current_term: 6 leader_uuid: "f7c202b562ce482db283463d7f02e136" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 04:01:52.211257 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.006s sys 0.046s
W20260806 04:01:52.226233 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.606s user 0.006s sys 0.047s
I20260806 04:01:52.224370 1137 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
W20260806 04:01:52.226547 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 606934us
I20260806 04:01:52.221506 5422 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:52.218653 4789 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:52.241078 4855 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:52.243592 32553 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.244143 5268 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:52.225036 5380 consensus_queue.cc:237] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.244644 4784 raft_consensus.cc:3037] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:52.246487 4486 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.240020 5421 consensus_queue.cc:1046] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.010s
I20260806 04:01:52.248979 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.253260 4523 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.253952 4481 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:52.240152 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2466) took 1354 ms. Trace:
I20260806 04:01:52.283039 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:52.329805 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 139616us
I20260806 04:01:52.319442 32614 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:52.364315 32738 rpcz_store.cc:276] 0806 04:01:50.885497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.534645 (+649148us) service_pool.cc:227] Handling call
0806 04:01:51.534980 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:52.107778 (+572798us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.239858 (+132080us) spinlock_profiling.cc:243] Waited 558 ms on lock 0x7b5400212ad4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:52.240062 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1283983104}
W20260806 04:01:52.363020 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 119499us
I20260806 04:01:52.366353 32553 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.259310 32615 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:52.260465 32603 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:52.234815 4875 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:52.371721 32604 raft_consensus.cc:1226] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
W20260806 04:01:52.262388 32474 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.015s sys 0.044s
W20260806 04:01:52.263267 5110 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.014s sys 0.020s
I20260806 04:01:52.267198 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:52.271744 32740 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.017s sys 0.048s
I20260806 04:01:52.272809 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:52.273754 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:52.372383 32474 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s user 0.015s sys 0.044s
I20260806 04:01:52.374542 32614 raft_consensus.cc:1226] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:52.264679 5298 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:52.375645 32740 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s user 0.017s sys 0.048s
I20260806 04:01:52.376252 32553 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:52.270710 5032 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.386458 32475 raft_consensus.cc:2379] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 4: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:52.351118 32427 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.389045 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:52.361259 5221 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.008s sys 0.028s
I20260806 04:01:52.268443 5136 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:52.386595 4915 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.024s sys 0.018s
I20260806 04:01:52.389482 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.288400 4679 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.333647 4637 consensus_queue.cc:1046] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.126s
W20260806 04:01:52.373970 5110 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.014s sys 0.020s
W20260806 04:01:52.420696 5221 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.620s user 0.008s sys 0.029s
I20260806 04:01:52.274549 5422 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:52.443665 32609 raft_consensus.cc:2396] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 04:01:52.440562 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:52.437757 32466 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.004s sys 0.009s
I20260806 04:01:52.451423 32682 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
W20260806 04:01:52.527587 32466 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s user 0.011s sys 0.011s
W20260806 04:01:52.438421 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 62246.6us
I20260806 04:01:52.513674 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.449388 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1781) took 1750 ms. Trace:
I20260806 04:01:52.391150 32473 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 11: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:52.451471 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 110898us
I20260806 04:01:52.481041 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 673) took 1647 ms. Trace:
I20260806 04:01:52.374202 5135 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:01:52.470855 32602 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:52.446664 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59498 (request call id 1787) took 1281 ms. Trace:
I20260806 04:01:52.371135 4523 raft_consensus.cc:2726] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 04:01:52.494905 4915 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.698s user 0.024s sys 0.019s
I20260806 04:01:52.288774 5337 raft_consensus.cc:3032] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:52.496070 4789 raft_consensus.cc:2781] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:01:52.533092 32743 raft_consensus.cc:1226] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [term 11 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 10. Current term is 11. Ops: []
I20260806 04:01:52.289526 32326 catalog_manager.cc:5495] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 reported cstate change: config changed from index 4 to 3, term changed from 6 to 4, leader changed from <none> to dd7344d85a154539b7f8c484f2a76772 (127.29.237.130), d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from VOTER to NON_VOTER, now has a pending config: VOTER ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129), VOTER dd7344d85a154539b7f8c484f2a76772 (127.29.237.130), VOTER f7c202b562ce482db283463d7f02e136 (127.29.237.131), VOTER d4a238f055a44f459c62c224daa4d541 (127.29.237.132). New cstate: current_term: 4 leader_uuid: "dd7344d85a154539b7f8c484f2a76772" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: NON_VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: true } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:52.530901 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.449355 5163 raft_consensus.cc:3032] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 LEADER]: Stepping down as leader of term 10
I20260806 04:01:52.454100 4486 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.530706 32474 rpcz_store.cc:276] 0806 04:01:51.164873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.690921 (+526048us) service_pool.cc:227] Handling call
0806 04:01:51.705121 (+ 14200us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.716035 (+ 10914us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 04:01:51.716410 (+ 375us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.388912 (+672502us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.439809 (+ 50897us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b18004e30a0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a32c638 00007fd16a32fb11 00007fd16a32fac9 00007fd16a32fa59 00007fd16a32fa21 00007fd16a32ed1d 00007fd173590204 00007fd173590039 00007fd16a30edc6
0806 04:01:52.446148 (+ 6339us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:52.446208 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:52.446274 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.446425 (+ 151us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.446591 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56968,"prepare.queue_time_us":2857,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":74,"raft.queue_time_us":104159,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":196,"spinlock_wait_cycles":38888320,"thread_start_us":658214,"threads_started":1}
W20260806 04:01:52.503995 32612 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.014s sys 0.002s
W20260806 04:01:52.555550 32612 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.745s user 0.014s sys 0.003s
I20260806 04:01:52.447392 4949 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:01:52.548952 32748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.020s sys 0.034s
I20260806 04:01:52.550031 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 886) took 2834 ms. Trace:
W20260806 04:01:52.561291 32748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.021s sys 0.034s
I20260806 04:01:52.513064 5221 raft_consensus.cc:686] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:52.389724 5268 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:52.540501 4915 raft_consensus.cc:686] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [term 5 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
W20260806 04:01:52.509675 4644 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.713s user 0.006s sys 0.011s
I20260806 04:01:52.556373 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 759) took 2084 ms. Trace:
I20260806 04:01:52.532452 5032 raft_consensus.cc:2741] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 04:01:52.540496 5337 raft_consensus.cc:729] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:52.529428 32467 rpcz_store.cc:276] 0806 04:01:50.699059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.563242 (+864183us) service_pool.cc:227] Handling call
0806 04:01:51.612313 (+ 49071us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:51.694331 (+ 82018us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.694357 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:51.694940 (+ 583us) log.cc:842] Serialized 52 byte log entry
0806 04:01:52.050607 (+355667us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:52.050648 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.050718 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:52.449160 (+398442us) raft_consensus.cc:1671] finished
0806 04:01:52.449182 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.449323 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13610,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"thread_start_us":355032,"threads_started":1,"wal-append.queue_time_us":695608}
I20260806 04:01:52.551146 5163 raft_consensus.cc:729] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:52.565023 32427 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:52.565747 4922 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:52.420475 5298 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:52.531287 32740 rpcz_store.cc:276] 0806 04:01:50.834100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.519721 (+685621us) service_pool.cc:227] Handling call
0806 04:01:51.529560 (+ 9839us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.529784 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 04:01:51.724476 (+194692us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.421804 (+697328us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:52.480697 (+ 58893us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:52.480753 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.480846 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.480982 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":556020,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":65464,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":13167,"thread_start_us":829267,"threads_started":2}
I20260806 04:01:52.562242 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 670) took 2887 ms. Trace:
I20260806 04:01:52.550426 5244 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:52.508486 5136 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.592492 32612 rpcz_store.cc:276] 0806 04:01:50.472201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.036738 (+564537us) service_pool.cc:227] Handling call
0806 04:01:51.125008 (+ 88270us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:51.231638 (+106630us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:51.327311 (+ 95673us) log.cc:842] Serialized 14 byte log entry
0806 04:01:51.766399 (+439088us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:51.809464 (+ 43065us) log.cc:842] Serialized 26 byte log entry
0806 04:01:51.809613 (+ 149us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:51.809873 (+ 260us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.555885 (+746012us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.555961 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:52.556008 (+ 47us) raft_consensus.cc:1671] finished
0806 04:01:52.556021 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.556280 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50869,"prepare.queue_time_us":79577,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":95,"raft.queue_time_us":25544,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":208,"thread_start_us":1092483,"threads_started":2,"wal-append.queue_time_us":757083}
I20260806 04:01:52.575453 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2474) took 1572 ms. Trace:
I20260806 04:01:52.563429 4849 raft_consensus.cc:2726] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975) with higher term. Message: Invalid argument: T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5.
I20260806 04:01:52.638445 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.561333 32466 rpcz_store.cc:276] 0806 04:01:49.715308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:50.352854 (+637546us) service_pool.cc:227] Handling call
0806 04:01:50.353086 (+ 232us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:50.353261 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 04:01:50.382063 (+ 28802us) log.cc:842] Serialized 14 byte log entry
0806 04:01:51.818134 (+1436071us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b1800301d80. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16cee382c 00007fd16a30e9dc 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a329155 00007fd16d24c3d0 00007fd16d25e576 00007fd16d26f0d7
0806 04:01:51.818553 (+ 419us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.549544 (+730991us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:52.549616 (+ 72us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:52.549682 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.549785 (+ 103us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.549959 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148039,"prepare.queue_time_us":29778,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":90,"raft.queue_time_us":114582,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":285,"spinlock_wait_cycles":33572864,"thread_start_us":817762,"threads_started":3,"wal-append.queue_time_us":46398,"wal-append.run_cpu_time_us":1168,"wal-append.run_wall_time_us":1073773}
I20260806 04:01:52.555105 4789 raft_consensus.cc:686] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [term 3 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
W20260806 04:01:52.601605 4644 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.785s user 0.007s sys 0.012s
W20260806 04:01:52.641829 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 76910.4us
I20260806 04:01:52.565150 4637 consensus_queue.cc:1046] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260806 04:01:52.645841 32738 raft_consensus.cc:1226] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:52.647600 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:52.648380 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 69573.6us
I20260806 04:01:52.641620 5244 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:52.643154 32748 rpcz_store.cc:276] 0806 04:01:49.675314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.286520 (+1611206us) service_pool.cc:227] Handling call
0806 04:01:51.367834 (+ 81314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.414359 (+ 46525us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 04:01:51.802795 (+388436us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.561890 (+759095us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:52.561944 (+ 54us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:52.561996 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.562068 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.562186 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8899,"prepare.queue_time_us":621448,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":2544,"raft.queue_time_us":499,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":102,"thread_start_us":942906,"threads_started":2}
I20260806 04:01:52.642235 32743 rpcz_store.cc:276] 0806 04:01:51.002714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.627988 (+625274us) service_pool.cc:227] Handling call
0806 04:01:51.642689 (+ 14701us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:52.554754 (+912065us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.575171 (+ 20417us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b54000f9e54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:52.575386 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2033075968}
I20260806 04:01:52.601816 4486 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:52.656760 4644 raft_consensus.cc:686] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
W20260806 04:01:52.661566 1136 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.018s sys 0.059s
W20260806 04:01:52.662587 1136 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.699s user 0.018s sys 0.059s
W20260806 04:01:52.661172 1188 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.012s sys 0.035s
W20260806 04:01:52.663012 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 705565us
W20260806 04:01:52.662039 1185 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.046s sys 0.031s
W20260806 04:01:52.664273 1185 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s user 0.046s sys 0.031s
I20260806 04:01:52.646478 4949 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 04:01:52.664003 1188 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s user 0.012s sys 0.035s
I20260806 04:01:52.666751 32428 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:52.657792 4679 raft_consensus.cc:3032] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 04:01:52.674055 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 253830us
W20260806 04:01:52.681818 32470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.044s sys 0.035s
W20260806 04:01:52.682775 32470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.824s user 0.044s sys 0.035s
I20260806 04:01:52.681751 1136 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:52.667112 4949 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:52.673313 4679 raft_consensus.cc:729] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:52.690044 32470 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 905) took 1032 ms. Trace:
W20260806 04:01:52.685602 32471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.000s sys 0.033s
W20260806 04:01:52.690819 32471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s user 0.000s sys 0.034s
I20260806 04:01:52.693390 32682 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:52.673053 5163 consensus_queue.cc:260] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.664980 4486 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.683961 5421 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:52.686786 4855 raft_consensus.cc:2781] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:52.690390 32470 rpcz_store.cc:276] 0806 04:01:51.658089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.835370 (+177281us) service_pool.cc:227] Handling call
0806 04:01:51.835724 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:51.835946 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:51.858453 (+ 22507us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.685174 (+826721us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:52.685225 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:52.689752 (+ 4527us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.689864 (+ 112us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.689989 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1984,"prepare.queue_time_us":5189,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"thread_start_us":824315,"threads_started":1}
I20260806 04:01:52.686708 4949 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:52.690186 4922 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:52.700937 5008 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.009s sys 0.029s
I20260806 04:01:52.714563 5421 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:52.719389 5434 log.cc:924] Time spent T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20260806 04:01:52.721149 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 44806.3us
I20260806 04:01:52.669301 5337 consensus_queue.cc:260] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:52.718230 5008 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.566s user 0.010s sys 0.029s
I20260806 04:01:52.716674 4875 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:52.726370 5136 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:52.721596 5190 consensus_meta.cc:287] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.539s user 0.011s sys 0.000s
W20260806 04:01:52.724193 32683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.015s sys 0.036s
W20260806 04:01:52.730726 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 62504.9us
W20260806 04:01:52.739874 32683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.625s user 0.015s sys 0.036s
W20260806 04:01:52.740742 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 626289us
I20260806 04:01:52.718041 4922 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:52.716343 4784 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:52.688262 4915 consensus_queue.cc:237] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.741828 1136 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:52.741483 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 245974us
I20260806 04:01:52.720793 4855 raft_consensus.cc:686] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:52.730237 5189 consensus_meta.cc:287] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.637s user 0.008s sys 0.004s
I20260806 04:01:52.701531 4481 raft_consensus.cc:686] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:52.689903 5221 consensus_queue.cc:237] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:52.749465 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 494410us
I20260806 04:01:52.754884 5443 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:52.756945 5445 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:52.760382 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2106) took 1737 ms. Trace:
I20260806 04:01:52.742569 5163 raft_consensus.cc:3037] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:52.731084 5268 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:52.762987 32427 connection.cc:232] RPC call timeout handler was delayed by -2.0793378560338169s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
W20260806 04:01:52.769918 32427 connection.cc:232] RPC call timeout handler was delayed by -2.016426406102255s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
I20260806 04:01:52.720408 4644 consensus_queue.cc:237] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:52.772173 32682 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:52.764398 32478 rpcz_store.cc:276] 0806 04:01:51.011454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.687563 (+676109us) service_pool.cc:227] Handling call
0806 04:01:51.694666 (+ 7103us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:51.694979 (+ 313us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.695012 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:51.695516 (+ 504us) log.cc:842] Serialized 26 byte log entry
0806 04:01:52.095332 (+399816us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:52.095369 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.095441 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:52.749009 (+653568us) raft_consensus.cc:1671] finished
0806 04:01:52.749027 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:52.749161 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13130,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":399431,"threads_started":1,"wal-append.queue_time_us":855113}
I20260806 04:01:52.776286 5443 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.776880 1136 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:52.778862 4993 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:52.749410 5298 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:52.758404 4875 raft_consensus.cc:686] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:52.730875 4789 consensus_queue.cc:237] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:52.779539 4993 raft_consensus.cc:2726] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:01:52.821877 4993 consensus_queue.cc:1035] T 9befcec6c17046459b03fa9ce98de94e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:01:52.822155 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.778877 5443 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.792860 5464 raft_consensus.cc:2781] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:52.823566 5464 raft_consensus.cc:686] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:52.793340 5430 consensus_queue.cc:1046] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260806 04:01:52.825624 5464 consensus_queue.cc:237] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.826310 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.827050 32749 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:52.797021 4855 consensus_queue.cc:237] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.798807 4481 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.801919 5478 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:52.803062 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.803555 5163 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:52.804731 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.807413 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.730849 4679 consensus_queue.cc:260] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.809247 5442 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:52.810024 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:52.812460 5438 log.cc:924] Time spent T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20260806 04:01:52.812704 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.813407 4486 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.816443 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.742690 5190 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:52.819239 5441 log.cc:924] Time spent T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20260806 04:01:52.819092 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:52.819542 5463 consensus_queue.cc:1035] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.077s
I20260806 04:01:52.780866 5412 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:52.748884 4784 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.775444 5268 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:52.842597 32553 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:52.845203 5244 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.760881 5189 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.852841 32738 raft_consensus.cc:2379] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:52.883170 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 100274us
W20260806 04:01:52.863412 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 114546us
W20260806 04:01:52.870205 5437 log.cc:924] Time spent T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.162s user 0.000s sys 0.001s
I20260806 04:01:52.862419 5275 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:01:52.866050 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 87635.2us
I20260806 04:01:52.870348 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f3a6d6cef24470a69dbafa39b3d3e1" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:52.860226 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 193532us
I20260806 04:01:52.881084 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.773871 5445 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:52.870790 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.890801 32553 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.863937 5421 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:52.891957 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:52.893673 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 92490.9us
I20260806 04:01:52.897516 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.897910 32473 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260806 04:01:52.884541 5298 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:52.902525 32740 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 9: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:52.906301 1136 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:52.901088 32553 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.905247 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:52.911710 32470 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:52.895926 5443 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 04:01:52.916990 1137 connection.cc:232] RPC call timeout handler was delayed by -1.2266738070175052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
I20260806 04:01:52.917758 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:52.902864 5252 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.005s sys 0.010s
I20260806 04:01:52.894726 5444 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:52.924539 5470 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:52.910907 4875 consensus_queue.cc:237] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:52.925473 5467 log.cc:924] Time spent T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 04:01:52.925928 1136 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:52.902807 5298 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:52.929900 5252 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s user 0.006s sys 0.010s
I20260806 04:01:52.927331 32737 raft_consensus.cc:1226] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:52.872010 5478 consensus_queue.cc:1046] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260806 04:01:52.917109 5221 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:52.895974 5268 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:52.967981 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 67716.3us
W20260806 04:01:52.971565 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 45737us
I20260806 04:01:52.925475 5445 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:52.930053 4789 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:52.972268 5488 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:52.980321 32749 raft_consensus.cc:1226] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:52.914552 5008 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260806 04:01:52.955212 5443 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:52.982578 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.982751 32553 leader_election.cc:302] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:52.984026 32427 connection.cc:232] RPC call timeout handler was delayed by -1.6678007980808616s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2601s for a response
W20260806 04:01:52.984548 32427 connection.cc:232] RPC call timeout handler was delayed by -1.4459043520037085s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17450000000000002s for a response
I20260806 04:01:52.985524 1136 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:52.985672 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:52.985908 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.986253 32478 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:52.886696 5442 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:52.991678 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:52.992506 5443 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:52.986415 5444 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:52.988320 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:52.993623 32604 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
W20260806 04:01:52.985548 5454 log.cc:924] Time spent T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
I20260806 04:01:52.994215 5442 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.994803 5444 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:52.990191 5455 log.cc:924] Time spent T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260806 04:01:52.983243 32602 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:52.995783 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.990305 5268 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.990899 5190 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:52.964730 4929 consensus_queue.cc:1046] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260806 04:01:52.988380 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:52.986667 32609 raft_consensus.cc:2379] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:52.989961 1136 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:52.929415 5380 consensus_queue.cc:1046] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260806 04:01:52.990749 5221 raft_consensus.cc:2726] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:52.963610 5298 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:52.988919 32612 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 04:01:52.987035 5049 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.785s user 0.010s sys 0.002s
I20260806 04:01:53.012964 32470 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:53.015340 32478 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
W20260806 04:01:53.018988 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 90084.8us
I20260806 04:01:52.992430 5469 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:53.024024 32553 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:53.025866 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 67137.3us
I20260806 04:01:53.029979 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:53.023392 5187 consensus_meta.cc:287] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.831s user 0.013s sys 0.001s
I20260806 04:01:53.012207 4855 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:53.030618 32737 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:53.032850 32682 leader_election.cc:398] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:53.033319 32683 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:53.033365 32682 leader_election.cc:401] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 04:01:53.034945 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:53.022140 5478 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:53.033136 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 855) took 1109 ms. Trace:
I20260806 04:01:53.041828 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:53.042410 5500 raft_consensus.cc:2781] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:53.043035 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:53.092651 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 61608.4us
I20260806 04:01:52.956969 5470 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.045976 32553 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:53.045167 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 55325.5us
I20260806 04:01:53.004119 4789 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:53.115648 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 69764.5us
I20260806 04:01:53.034194 4784 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:53.033515 5463 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:53.123893 1136 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.036264 5187 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:53.023013 5049 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.041667 5484 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:53.115731 4929 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election won for term 8
W20260806 04:01:53.140669 32551 leader_election.cc:334] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 1.574s (SENT)
I20260806 04:01:53.141243 32551 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:53.144380 32467 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 10. Current term is 11. Ops: []
I20260806 04:01:53.129626 5500 raft_consensus.cc:686] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [term 8 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:53.093098 5478 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:53.184567 5478 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:53.041713 4855 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.187103 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:53.130482 4784 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:53.156404 5501 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:53.156457 5473 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:53.134956 5463 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.170709 4789 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.145993 4929 raft_consensus.cc:686] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:53.181496 5049 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.182256 5484 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.183418 5221 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 04:01:53.132258 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 124418us
W20260806 04:01:53.183625 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 46467.1us
I20260806 04:01:53.189465 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:53.113032 1188 rpcz_store.cc:276] 0806 04:01:51.923685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:51.924674 (+ 989us) service_pool.cc:227] Handling call
0806 04:01:51.948224 (+ 23550us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:51.963431 (+ 15207us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:51.963464 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:51.994336 (+ 30872us) log.cc:842] Serialized 52 byte log entry
0806 04:01:52.667394 (+673058us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:52.832600 (+165206us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.836220 (+ 3620us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.836394 (+ 174us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:52.836473 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:53.030771 (+194298us) raft_consensus.cc:1671] finished
0806 04:01:53.030790 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:53.033076 (+ 2286us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31837,"prepare.queue_time_us":323726,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":130157,"raft.run_cpu_time_us":500,"raft.run_wall_time_us":38470,"thread_start_us":816440,"threads_started":2,"wal-append.queue_time_us":832486}
I20260806 04:01:53.250480 32473 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
W20260806 04:01:53.241631 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 114522us
I20260806 04:01:53.179600 5298 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.224406 5501 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:53.175611 4922 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.147651 5469 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260806 04:01:53.222851 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 73530.6us
I20260806 04:01:53.251416 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:53.233539 5189 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.232481 4784 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.293672 32473 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:53.294030 5519 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:53.248255 5444 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.294747 4784 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:53.295992 5444 consensus_queue.cc:1046] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:53.297538 5444 raft_consensus.cc:3032] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:53.297976 5444 raft_consensus.cc:729] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:53.301263 32682 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:53.252648 5486 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:53.304627 5486 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.304821 4929 consensus_queue.cc:237] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:53.305604 4922 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:53.306635 32551 leader_election.cc:334] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 1.764s (SENT)
I20260806 04:01:53.306972 5486 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.331869 1136 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:53.330178 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 237733us
I20260806 04:01:53.283386 5501 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:53.287499 5221 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:53.282788 5508 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:53.333621 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.318012 5049 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:53.341768 32553 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:53.343616 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 119196us
I20260806 04:01:53.335232 5221 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:53.345022 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:53.345497 5221 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:53.345861 32473 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:53.338611 5508 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.349329 5298 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.344712 4784 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:53.350203 5508 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:53.342396 5470 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:53.351239 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 47051.8us
I20260806 04:01:53.352173 1136 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.352385 5484 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:53.352885 4949 consensus_meta.cc:287] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.635s user 0.011s sys 0.000s
I20260806 04:01:53.353314 4949 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.353499 32473 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:53.353982 1136 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:53.335870 5500 consensus_queue.cc:237] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.360899 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:53.362394 32683 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:53.363648 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 62488.5us
I20260806 04:01:53.363267 32737 raft_consensus.cc:2379] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 5: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:53.351424 5475 log.cc:924] Time spent T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.226s user 0.000s sys 0.001s
I20260806 04:01:53.360236 4644 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:01:53.407106 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 53201.2us
I20260806 04:01:53.408351 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:53.362864 5508 raft_consensus.cc:2741] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260806 04:01:53.363690 5298 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:53.363374 5445 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:53.415925 4849 consensus_queue.cc:1046] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 04:01:53.298913 5444 consensus_queue.cc:260] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:53.427001 1137 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.373651 5470 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 04:01:53.436546 32551 leader_election.cc:334] T 0bc7e4a68014409c8199e1e003add00a P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181): Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 1.855s (SENT)
I20260806 04:01:53.442785 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.445958 4849 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:53.446236 5532 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:01:53.447394 5505 log.cc:924] Time spent T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20260806 04:01:53.451128 4849 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:53.451567 4849 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:53.451790 1137 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:53.451939 4849 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:53.453116 32749 raft_consensus.cc:1261] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:01:53.453384 5495 raft_consensus.cc:2781] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:53.453910 5495 raft_consensus.cc:686] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:53.454825 5495 consensus_queue.cc:237] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.456640 5541 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:53.457186 5541 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.458323 32473 raft_consensus.cc:1261] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:01:53.459465 5541 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.460441 4849 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:53.462000 5516 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20260806 04:01:53.462809 4849 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:53.416203 5463 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.460373 5486 consensus_queue.cc:1035] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260806 04:01:53.367372 5049 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.484267 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:53.484493 32466 raft_consensus.cc:3037] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:01:53.485319 32748 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:53.486526 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:53.487246 32738 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 9.
I20260806 04:01:53.487718 5221 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.488494 32746 raft_consensus.cc:1226] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 04:01:53.446046 5522 consensus_queue.cc:1046] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.146s
I20260806 04:01:53.490011 5190 consensus_queue.cc:1035] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 04:01:53.491205 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:53.435276 4855 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.492257 32473 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
W20260806 04:01:53.493865 5422 consensus_meta.cc:287] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.931s user 0.011s sys 0.000s
I20260806 04:01:53.416495 5445 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.496438 5445 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.497129 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:53.497751 4949 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.498481 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2114) took 1421 ms. Trace:
I20260806 04:01:53.498746 32682 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:53.499719 5049 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.500597 32682 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:53.488776 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:53.501680 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:53.501904 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:53.502478 32610 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 2219) took 1451 ms (client timeout 1909 ms). Trace:
I20260806 04:01:53.500032 5478 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:53.502848 32610 rpcz_store.cc:269] 0806 04:01:52.051062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:52.056160 (+ 5098us) service_pool.cc:227] Handling call
0806 04:01:53.502295 (+1446135us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003340d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:53.502424 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2876318976}
I20260806 04:01:53.503085 32682 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:53.504781 5478 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:53.504931 32682 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:53.526111 32737 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:53.529088 32553 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:53.530184 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:53.498818 32471 rpcz_store.cc:276] 0806 04:01:52.076952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:52.122474 (+ 45522us) service_pool.cc:227] Handling call
0806 04:01:52.148357 (+ 25883us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:52.160977 (+ 12620us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 04:01:52.165495 (+ 4518us) log.cc:842] Serialized 14 byte log entry
0806 04:01:52.839304 (+673809us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:01:52.839359 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:52.839405 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:53.498265 (+658860us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b540019a354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:53.498312 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:53.498427 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1427,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":109478,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":165,"spinlock_wait_cycles":28458368,"thread_start_us":673347,"threads_started":2,"wal-append.queue_time_us":598582}
I20260806 04:01:53.535266 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:53.536362 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:53.503402 5527 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:01:53.542177 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:53.540989 32748 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:53.546070 32746 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:53.548681 32745 raft_consensus.cc:1261] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:01:53.549510 5543 raft_consensus.cc:2781] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:53.539732 5486 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:53.560106 5486 raft_consensus.cc:686] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:53.561066 5486 consensus_queue.cc:237] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.453136 4481 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.087s
I20260806 04:01:53.565646 1137 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:53.568696 5508 consensus_queue.cc:1046] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:53.571456 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:53.531970 5519 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 04:01:53.547672 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 111154us
I20260806 04:01:53.529381 5532 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:53.580737 32466 raft_consensus.cc:1261] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 04:01:53.583505 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.542570 5341 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:53.575672 32473 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:53.587990 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.592949 32474 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:53.593452 32553 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:53.597263 32466 raft_consensus.cc:1226] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 04:01:53.598978 32470 raft_consensus.cc:1226] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 04:01:53.574944 5221 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.578511 4849 consensus_queue.cc:1046] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260806 04:01:53.610168 4849 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:53.610813 4849 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:53.611400 4849 raft_consensus.cc:3032] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:53.611753 4849 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:53.612773 32471 raft_consensus.cc:3032] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:53.613159 32471 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:53.613888 32471 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.614814 32471 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:53.608994 5469 consensus_queue.cc:1046] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260806 04:01:53.618360 5488 raft_consensus.cc:3032] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:53.619392 32470 raft_consensus.cc:1226] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:53.618618 4849 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.597991 5445 consensus_queue.cc:1035] T bec6dcc3e9784bbe83c0ee7452223cb4 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260806 04:01:53.606482 5544 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:53.584733 5532 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:53.525656 5422 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.595917 4637 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:53.608961 4523 raft_consensus.cc:2781] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:53.636942 5221 consensus_queue.cc:1046] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:01:53.638114 5221 raft_consensus.cc:3032] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:53.638444 5221 raft_consensus.cc:729] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:53.638692 5244 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.639254 5221 consensus_queue.cc:260] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.640754 4637 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.643448 4637 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:53.643615 5504 log.cc:924] Time spent T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.601s user 0.000s sys 0.004s
I20260806 04:01:53.644536 4523 raft_consensus.cc:686] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:53.645529 4523 consensus_queue.cc:237] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:53.662070 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 157198us
I20260806 04:01:53.585140 5478 consensus_queue.cc:1046] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260806 04:01:53.665998 32469 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 9: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:53.591513 4481 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260806 04:01:53.632033 5544 raft_consensus.cc:482] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:53.644474 4849 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:53.628835 5488 raft_consensus.cc:729] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
W20260806 04:01:53.671823 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 137047us
W20260806 04:01:53.674680 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 307763us
W20260806 04:01:53.683607 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 124380us
I20260806 04:01:53.634804 5532 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:53.663317 5221 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:53.609103 5501 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.698045 32553 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:53.699800 1136 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.668803 5189 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:53.682765 5544 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Advancing to term 10
W20260806 04:01:53.720470 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 132550us
I20260806 04:01:53.725060 32427 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:53.692850 5473 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:53.725592 5527 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:53.731735 32682 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:53.699622 5543 raft_consensus.cc:686] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:53.733613 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:53.734263 1188 raft_consensus.cc:2362] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6. Re-sending same reply.
I20260806 04:01:53.712492 5445 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:53.716323 4915 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:53.728590 5495 raft_consensus.cc:2726] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:53.729787 5519 raft_consensus.cc:686] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:53.804457 5554 log.cc:924] Time spent T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260806 04:01:53.749325 4915 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:53.720412 5268 consensus_queue.cc:1035] T db915af860484020942e9495eda28804 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260806 04:01:53.980480 5584 consensus_queue.cc:1046] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
I20260806 04:01:53.807276 5488 consensus_queue.cc:260] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.749699 5473 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.885589 5445 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:54.106351 32608 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41308 (request call id 2217) took 2071 ms (client timeout 2352 ms). Trace:
W20260806 04:01:54.106976 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 375299us
I20260806 04:01:54.109598 5422 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:53.803237 5527 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:53.996508 5543 consensus_queue.cc:237] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.111487 5501 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:54.112002 32427 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:54.113524 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.127913 4915 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.157423 32682 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:54.114744 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.114991 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.115648 32740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33978 (request call id 1882) took 1302 ms. Trace:
I20260806 04:01:53.803355 5551 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:54.117628 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.119665 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.119707 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.158504 32740 rpcz_store.cc:276] 0806 04:01:52.813512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:52.816152 (+ 2640us) service_pool.cc:227] Handling call
0806 04:01:53.753966 (+937814us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b54001f3454. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:54.115549 (+361583us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108584832}
I20260806 04:01:54.174961 1190 raft_consensus.cc:2362] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5. Re-sending same reply.
W20260806 04:01:54.124714 32471 consensus_meta.cc:287] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.510s user 0.011s sys 0.002s
I20260806 04:01:54.177517 1183 raft_consensus.cc:2379] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 8: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:54.130473 1192 raft_consensus.cc:1261] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260806 04:01:54.137959 32608 rpcz_store.cc:269] 0806 04:01:52.035288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:52.035590 (+ 302us) service_pool.cc:227] Handling call
0806 04:01:53.754716 (+1719126us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x7b5400086954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:54.106280 (+351564us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1974392064}
I20260806 04:01:54.143879 32477 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 04:01:54.114466 5468 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 1.082s user 0.008s sys 0.004s
I20260806 04:01:54.131475 4637 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.158703 1189 raft_consensus.cc:2396] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:01:54.173246 1186 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:54.187287 32471 raft_consensus.cc:1261] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:54.194962 5580 raft_consensus.cc:3032] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:54.195469 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.195427 5580 raft_consensus.cc:729] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:54.196360 1188 raft_consensus.cc:2396] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 04:01:54.199329 32466 raft_consensus.cc:1261] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260806 04:01:54.196588 5473 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:54.135154 5478 consensus_queue.cc:1046] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260806 04:01:54.199723 32466 pending_rounds.cc:76] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 4
I20260806 04:01:54.136217 5500 consensus_queue.cc:1046] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.050s
I20260806 04:01:54.200536 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:54.208150 32739 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
W20260806 04:01:54.201645 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 89754.6us
I20260806 04:01:54.203979 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.209429 1183 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 11: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:54.199519 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.203068 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.181821 5422 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:54.210665 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.207602 32466 pending_rounds.cc:97] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260806 04:01:54.215598 32466 raft_consensus.cc:2866] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:54.212500 1186 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
I20260806 04:01:54.217238 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.213709 1192 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:54.222324 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.223038 1188 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:54.223893 1189 raft_consensus.cc:2379] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:54.228487 5500 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.229022 5268 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.229149 5500 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.229701 5268 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.231828 5268 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:54.216826 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 81815.4us
W20260806 04:01:54.238776 5423 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260806 04:01:54.181984 5532 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.240796 5532 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:54.221858 5221 consensus_meta.cc:287] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.502s user 0.010s sys 0.000s
I20260806 04:01:54.245440 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.244323 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.242797 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.214551 5478 consensus_queue.cc:1225] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Peer ae0b3e43e7a64f789e51018fd5518d23 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260806 04:01:54.254241 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.254463 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.250005 1183 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:54.232419 5268 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 LEADER]: Stepping down as leader of term 10
I20260806 04:01:54.190814 4637 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.263327 32427 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.264269 5560 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:54.264699 5560 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:54.265058 5560 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:54.265690 4637 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:54.265868 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:54.266536 32466 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 10.
I20260806 04:01:54.261230 1188 raft_consensus.cc:2379] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 9: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:54.262853 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.268764 1183 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:54.267583 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.269680 1192 raft_consensus.cc:3037] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 8
I20260806 04:01:54.270203 4679 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:54.232103 5500 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:54.273527 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:54.272007 5557 log.cc:924] Time spent T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.352s user 0.002s sys 0.004s
I20260806 04:01:54.245729 5508 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.282763 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:54.284018 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.293490 32471 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:54.238838 5596 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:54.289081 32739 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:54.293561 32740 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:54.305877 32748 raft_consensus.cc:1226] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:54.272264 5268 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
W20260806 04:01:54.313941 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 50684.5us
I20260806 04:01:54.318284 32427 leader_election.cc:302] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.316004 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.323943 32427 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.331089 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:54.310572 5500 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.322227 4637 raft_consensus.cc:2726] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 04:01:54.278815 5221 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20260806 04:01:54.331808 32471 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:54.342307 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.334282 5421 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 04:01:54.352141 5421 raft_consensus.cc:686] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:54.352615 5561 log.cc:924] Time spent T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260806 04:01:54.353161 5421 consensus_queue.cc:237] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.354079 5603 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:54.355247 1188 raft_consensus.cc:2396] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 04:01:54.276767 5519 consensus_queue.cc:237] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.349905 1137 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:54.351250 32427 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.357038 5548 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:54.357434 5548 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:54.357779 5548 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:54.358520 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:54.358693 5445 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:54.359073 32427 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.359861 4637 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:54.360584 32427 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.360237 4637 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:54.362226 4637 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:54.362635 5337 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.218290 5580 consensus_queue.cc:260] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.322966 5596 raft_consensus.cc:686] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:54.340616 5544 consensus_meta.cc:287] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.597s user 0.010s sys 0.000s
I20260806 04:01:54.312832 5508 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.374078 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:54.355252 5595 log.cc:924] Time spent T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
I20260806 04:01:54.369802 5445 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:54.377687 1188 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:54.323189 5470 consensus_queue.cc:1035] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260806 04:01:54.383140 5430 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.388257 5273 raft_consensus.cc:3032] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 04:01:54.388720 5273 raft_consensus.cc:729] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:54.389643 5273 consensus_queue.cc:260] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.391091 5273 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Advancing to term 4
I20260806 04:01:54.393867 5584 consensus_queue.cc:1046] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:54.380688 5581 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:54.395031 5581 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:54.395401 5581 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:54.398530 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.398802 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.404038 1192 raft_consensus.cc:2454] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:54.404292 32749 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:01:54.402561 1187 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:54.408450 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.350574 5500 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.409209 1183 raft_consensus.cc:2379] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:54.376909 5580 raft_consensus.cc:3037] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:54.410475 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.380905 5445 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:54.415218 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 257865us
W20260806 04:01:54.418530 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 237017us
I20260806 04:01:54.416479 32682 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:54.402153 4915 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.454155 1192 raft_consensus.cc:1261] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 04:01:54.455222 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.456061 1187 raft_consensus.cc:2379] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 5: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:54.377231 5544 raft_consensus.cc:504] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.415849 5445 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:54.423568 32553 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:54.381403 5268 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.500705 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.470878 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.472374 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.510654 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:54.490985 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.522373 32615 raft_consensus.cc:2379] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:54.490937 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:54.504710 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.522972 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.496474 32749 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:54.523114 32604 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:54.502285 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.523993 1186 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:54.500371 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.510284 32608 raft_consensus.cc:1261] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:54.492614 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.525578 1183 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:54.475394 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.527753 1187 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:54.528631 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.521519 32612 raft_consensus.cc:2454] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
W20260806 04:01:54.530876 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 107376us
I20260806 04:01:54.540998 32682 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:54.529220 1187 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
I20260806 04:01:54.545629 32609 raft_consensus.cc:2354] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:54.546880 1189 raft_consensus.cc:2379] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 10: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:54.547626 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.548427 32610 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:54.551416 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.552629 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:54.555299 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:54.555253 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.555996 32615 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.557152 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.457775 5610 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:54.559544 32477 raft_consensus.cc:2396] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260806 04:01:54.552134 1192 raft_consensus.cc:2454] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:54.555429 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:54.563203 32612 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:54.415490 5275 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.124s
I20260806 04:01:54.423708 5463 consensus_queue.cc:1046] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20260806 04:01:54.543304 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.513131 5500 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:54.564509 1186 raft_consensus.cc:2379] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 11: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:54.574716 32608 raft_consensus.cc:1226] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 3. Current term is 4. Ops: []
I20260806 04:01:54.565557 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.513936 5508 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:54.564926 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:54.566890 1137 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.564679 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8417acc05db341aca15704cd05736651" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:54.608731 32614 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:54.606269 32610 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 04:01:54.606473 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 46638.4us
I20260806 04:01:54.604878 32743 raft_consensus.cc:1226] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:54.607430 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.607932 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.610881 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:54.573307 1187 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
I20260806 04:01:54.614727 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.615248 1187 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:54.456858 5136 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.102s
I20260806 04:01:54.617175 5244 consensus_queue.cc:1035] T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:54.617654 5136 raft_consensus.cc:3032] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:54.618108 5136 raft_consensus.cc:729] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:54.618266 5519 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:54.618695 5519 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:54.618937 32682 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:54.619041 5519 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:54.619968 5527 raft_consensus.cc:2741] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 04:01:54.620532 4915 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:54.621464 32682 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:54.622252 5527 raft_consensus.cc:2781] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:54.622637 5527 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:54.622983 5527 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:54.624266 32615 raft_consensus.cc:2454] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:54.625396 5618 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:54.515563 5603 consensus_queue.cc:1035] T 65c8a38c9f554ef887e7782aa21ad891 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.113s
I20260806 04:01:54.628763 5478 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.629256 5478 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.629648 5519 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.629875 5560 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.631191 5478 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:54.631626 5519 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:54.632757 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.639369 32682 leader_election.cc:302] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:54.605988 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:54.566424 5486 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.961s user 0.017s sys 0.041s
I20260806 04:01:54.645664 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.635103 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:54.647608 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 80832.4us
I20260806 04:01:54.646761 1187 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:54.560595 5268 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:54.651751 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.562290 5610 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:54.565032 5577 raft_consensus.cc:2781] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:54.631879 5560 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:54.660807 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:54.569657 5470 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.661417 32739 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.652522 32604 raft_consensus.cc:2454] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:01:54.659993 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.662298 1187 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:54.576280 5501 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:54.660430 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.640472 1186 raft_consensus.cc:2362] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5. Re-sending same reply.
I20260806 04:01:54.670390 32473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:54.674296 5559 log.cc:924] Time spent T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.949s user 0.002s sys 0.016s
I20260806 04:01:54.674619 32475 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:01:54.685371 32467 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260806 04:01:54.686782 32682 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:54.685757 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:54.685703 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:54.684345 1136 connection.cc:232] RPC call timeout handler was delayed by -2.4778583829756826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
I20260806 04:01:54.687979 32604 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 10
I20260806 04:01:54.688819 1187 raft_consensus.cc:2396] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:01:54.688994 32746 raft_consensus.cc:1226] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:54.689193 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:54.695201 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c0e905630e43668379f2f95684775f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.695430 1137 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.691717 32551 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:54.655154 5508 raft_consensus.cc:2726] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:01:54.698787 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:54.698841 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.700278 1187 raft_consensus.cc:2379] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 8: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:54.645365 5486 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s user 0.019s sys 0.044s
I20260806 04:01:54.707162 5473 consensus_queue.cc:1046] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:01:54.695086 5501 raft_consensus.cc:686] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:54.674957 4679 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.708756 4679 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.619027 5136 consensus_queue.cc:260] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.709102 5501 consensus_queue.cc:237] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.710183 5136 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.710757 4679 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:54.661432 5577 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:54.712293 5577 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:54.574751 5578 consensus_queue.cc:1035] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.111s
I20260806 04:01:54.713763 5619 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:01:54.661808 5610 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:54.716336 4849 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.717227 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.702901 32604 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 10.
I20260806 04:01:54.718016 1187 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:54.668047 5519 raft_consensus.cc:2781] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:54.718384 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.664105 5551 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:54.703789 32739 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:54.723186 32614 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:54.569955 4637 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.725273 32604 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 9.
I20260806 04:01:54.726632 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.500974 5596 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.744906 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.750037 32477 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:54.716910 4849 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.751034 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.757066 1186 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.751235 1192 raft_consensus.cc:2454] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:54.708711 5430 raft_consensus.cc:2726] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:54.629657 5548 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:54.761124 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 69498us
I20260806 04:01:54.648394 5543 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.088s
I20260806 04:01:54.762499 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:54.766156 32604 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.764745 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:54.766513 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 405962us
I20260806 04:01:54.764112 32473 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:54.769069 1185 raft_consensus.cc:2396] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 04:01:54.768939 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.645174 5581 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.685986 5470 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.777485 1190 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:54.777576 32427 leader_election.cc:302] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.776293 32614 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:54.778892 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 50994us
I20260806 04:01:54.779577 32553 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.777362 32478 raft_consensus.cc:1226] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:54.752842 4915 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:54.717968 5604 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:54.812496 1187 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:54.813001 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305cbd07ce124048b7ac2f99b2aa40db" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:54.835155 32610 raft_consensus.cc:2396] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 9 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 04:01:54.766896 32738 raft_consensus.cc:2932] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
I20260806 04:01:54.777318 32551 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.814075 32427 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.815747 5548 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:54.752178 5519 raft_consensus.cc:686] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:54.829181 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.841967 1185 raft_consensus.cc:2379] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:54.842124 32749 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:54.818739 5430 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
W20260806 04:01:54.844422 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 67184.8us
I20260806 04:01:54.846199 32551 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.847281 32551 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:54.850303 5642 raft_consensus.cc:2781] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:54.850713 5642 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:54.851043 5642 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:54.854678 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.855406 5636 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.104s
I20260806 04:01:54.855818 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.856624 1187 raft_consensus.cc:2379] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:54.857432 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.858661 1187 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:54.860219 32743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 691) took 1384 ms. Trace:
I20260806 04:01:54.860582 32743 rpcz_store.cc:276] 0806 04:01:53.476154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:53.476284 (+ 130us) service_pool.cc:227] Handling call
0806 04:01:53.484367 (+ 8083us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:54.617873 (+1133506us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:54.859996 (+242123us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400212fd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:54.860167 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2576423424}
I20260806 04:01:54.753633 5486 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.819007 4679 raft_consensus.cc:2726] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:54.839560 5606 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:54.840233 4637 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:54.855031 5622 log.cc:924] Time spent T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
I20260806 04:01:54.841795 5519 consensus_queue.cc:237] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.857869 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:54.879741 1185 raft_consensus.cc:3037] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.868346 32551 leader_election.cc:302] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:54.821823 32553 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:54.860613 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.828203 32604 raft_consensus.cc:2454] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:54.867689 1188 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:54.868191 32612 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
W20260806 04:01:54.874089 32428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.511s user 0.030s sys 0.029s
W20260806 04:01:54.882129 32428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.519s user 0.031s sys 0.029s
W20260806 04:01:54.882467 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.51951e+06us
I20260806 04:01:54.827401 5221 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 04:01:54.901489 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 47463.6us
I20260806 04:01:54.902806 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.903352 32551 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.903600 32604 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:01:54.887051 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.888619 32610 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:54.830514 5620 raft_consensus.cc:2781] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:54.905778 1187 raft_consensus.cc:1261] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:01:54.909091 32551 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.909871 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.910965 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:54.899966 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 78260.1us
W20260806 04:01:54.899884 5623 log.cc:924] Time spent T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
W20260806 04:01:54.905797 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 91999.5us
I20260806 04:01:54.940529 1186 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:54.872936 5544 leader_election.cc:288] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:54.914723 32741 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:54.919413 5481 raft_consensus.cc:3032] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:54.911849 5636 raft_consensus.cc:2781] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:54.915638 5221 raft_consensus.cc:2726] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:54.923009 5610 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.907292 5618 raft_consensus.cc:2726] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:54.926189 5488 raft_consensus.cc:3037] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:54.926990 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:54.928486 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:54.929836 1187 raft_consensus.cc:1261] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:01:54.932195 5527 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.932597 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:54.936708 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:54.938241 5624 log.cc:924] Time spent T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260806 04:01:54.937772 1188 raft_consensus.cc:1261] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 04:01:54.960482 1187 raft_consensus.cc:1226] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:54.960525 32746 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 9: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
W20260806 04:01:54.962311 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 53305.6us
I20260806 04:01:54.956446 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:54.968742 32551 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:54.974594 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:54.974633 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 152477us
I20260806 04:01:54.978972 32741 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:54.978948 32551 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:54.945591 5481 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:54.945331 5620 raft_consensus.cc:686] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:54.877198 5486 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:54.983618 32737 raft_consensus.cc:1226] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260806 04:01:54.974133 5648 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:54.984970 32749 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:54.986272 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:54.987453 32427 connection.cc:232] RPC call timeout handler was delayed by -1.9595612639095634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.266s for a response
I20260806 04:01:54.948887 5645 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:54.988883 32427 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:54.961645 5636 raft_consensus.cc:686] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:54.989997 32746 raft_consensus.cc:1261] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 04:01:54.932546 5642 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:54.970816 5627 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 04:01:55.008109 32427 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:54.980773 4789 raft_consensus.cc:2726] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260806 04:01:55.008385 1183 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:54.998126 1185 raft_consensus.cc:2454] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
W20260806 04:01:55.063862 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 55880.4us
I20260806 04:01:55.069933 32427 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.070367 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 954) took 1335 ms. Trace:
I20260806 04:01:55.071342 32427 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.072265 5616 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:55.072674 5581 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.072827 5616 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.073642 5481 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.075091 5616 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:55.064419 5636 consensus_queue.cc:237] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.076432 5548 raft_consensus.cc:2726] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:55.077055 5548 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:55.063565 5610 leader_election.cc:288] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:55.062310 5642 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:55.079375 5606 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:55.030131 32741 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 10.
I20260806 04:01:55.018599 1187 raft_consensus.cc:3032] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:55.072858 32427 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.101182 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:55.101848 32475 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:55.081213 1187 raft_consensus.cc:729] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:55.082003 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:54.975845 5422 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260806 04:01:55.105831 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:55.082051 1137 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:55.079782 5606 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:55.113046 32739 raft_consensus.cc:1226] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:55.065857 5527 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:55.105268 5445 consensus_meta.cc:287] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.554s user 0.008s sys 0.001s
I20260806 04:01:55.067390 5645 raft_consensus.cc:2741] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 04:01:55.103358 1187 consensus_queue.cc:260] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.128921 1187 raft_consensus.cc:3037] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:55.111068 5430 raft_consensus.cc:2726] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:55.082580 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:55.140806 1185 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:55.117304 1183 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
I20260806 04:01:55.066987 5486 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.013352 5163 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Leader election won for term 11
I20260806 04:01:55.083390 5483 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:55.107470 5551 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:55.011878 5627 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:55.064550 5620 consensus_queue.cc:237] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.128340 5445 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:55.131263 5661 log.cc:924] Time spent T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.059s user 0.001s sys 0.001s
I20260806 04:01:55.131072 5633 consensus_queue.cc:1035] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260806 04:01:55.140861 1187 raft_consensus.cc:1261] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 04:01:55.142025 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
W20260806 04:01:55.145354 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 166469us
I20260806 04:01:55.123533 5606 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:55.123744 5581 raft_consensus.cc:2781] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader election won for term 10
W20260806 04:01:55.146847 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 190470us
I20260806 04:01:55.150820 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:55.164197 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:55.165105 1185 raft_consensus.cc:2454] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:54.995471 5648 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.143187 5470 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:55.156898 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:55.166229 32427 leader_election.cc:302] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:55.168218 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:55.169824 32615 raft_consensus.cc:1226] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:55.171267 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:55.171860 1187 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:55.172169 5380 raft_consensus.cc:2741] T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 04:01:55.172411 32474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 907) took 1665 ms. Trace:
I20260806 04:01:55.172863 32474 rpcz_store.cc:276] 0806 04:01:53.507147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:53.507553 (+ 406us) service_pool.cc:227] Handling call
0806 04:01:53.592764 (+ 85211us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:53.593826 (+ 1062us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:55.172203 (+1578377us) spinlock_profiling.cc:243] Waited 60.4 ms on lock 0x7b54001e05d0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:55.172347 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138932480}
W20260806 04:01:55.175792 32469 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1797) took 2645 ms (client timeout 1910 ms). Trace:
W20260806 04:01:55.176263 32469 rpcz_store.cc:269] 0806 04:01:52.530488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:52.530635 (+ 147us) service_pool.cc:227] Handling call
0806 04:01:55.175567 (+2644932us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001b1a54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:55.175710 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2570418048}
I20260806 04:01:55.176832 5648 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.181018 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:55.181720 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:55.070770 1184 rpcz_store.cc:276] 0806 04:01:53.735391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:53.740608 (+ 5217us) service_pool.cc:227] Handling call
0806 04:01:53.807467 (+ 66859us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:54.186482 (+379015us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:01:54.192593 (+ 6111us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:54.254749 (+ 62156us) log.cc:842] Serialized 26 byte log entry
0806 04:01:54.637528 (+382779us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:54.645873 (+ 8345us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:54.649502 (+ 3629us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:55.070122 (+420620us) raft_consensus.cc:1671] finished
0806 04:01:55.070141 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:55.070289 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25649,"raft.queue_time_us":33770,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11018,"thread_start_us":242140,"threads_started":1,"wal-append.queue_time_us":372818}
I20260806 04:01:55.182520 1185 raft_consensus.cc:2379] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 8: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:55.184037 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:55.186057 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:55.187292 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:55.169456 1183 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
I20260806 04:01:55.188338 1187 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 11.
I20260806 04:01:55.148842 5551 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:55.189853 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:55.190480 32551 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.201021 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:55.192489 5544 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:55.194079 32471 raft_consensus.cc:1226] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:55.150141 5627 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:55.198738 32427 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.201057 1185 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 9: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:55.150225 5163 raft_consensus.cc:686] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:55.208901 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:55.210379 1187 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 04:01:55.166733 5430 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:55.212738 32427 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:55.212602 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:55.208384 32428 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:55.213644 32427 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.154939 5577 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.217053 32469 raft_consensus.cc:2396] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:01:55.218572 1185 raft_consensus.cc:1226] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 04:01:55.183015 5581 raft_consensus.cc:686] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [term 10 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:55.225026 32551 leader_election.cc:302] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:55.206328 5551 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:55.236792 32471 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:01:55.217721 5444 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:55.219770 5380 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:55.217991 5648 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:55.220661 5430 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:55.243448 1185 raft_consensus.cc:1261] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:01:55.220418 5422 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Leader election won for term 11
I20260806 04:01:55.220357 5673 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:55.269239 1183 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:55.220021 5519 consensus_queue.cc:1046] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260806 04:01:55.269819 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.273713 5163 consensus_queue.cc:237] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.275413 5548 raft_consensus.cc:2726] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:55.276548 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:55.303635 32740 raft_consensus.cc:2454] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:55.276710 4486 raft_consensus.cc:3032] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:55.305004 4486 raft_consensus.cc:729] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:55.305868 4486 consensus_queue.cc:260] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.306881 4486 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:55.268502 5430 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:55.278203 5577 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:55.267025 5470 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:55.264496 5610 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.018s
I20260806 04:01:55.266541 5380 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:55.283476 5600 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:55.289011 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 965) took 1173 ms. Trace:
I20260806 04:01:55.287889 32475 raft_consensus.cc:2454] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:55.331760 1189 rpcz_store.cc:276] 0806 04:01:54.115779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.238099 (+122320us) service_pool.cc:227] Handling call
0806 04:01:55.288811 (+1050712us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54007a1a54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:55.288953 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":617622272}
I20260806 04:01:55.296780 4679 raft_consensus.cc:2726] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:01:55.266736 32551 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.213348 5544 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.302209 5486 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
W20260806 04:01:55.302769 5643 log.cc:924] Time spent T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.331s user 0.003s sys 0.000s
W20260806 04:01:55.316438 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 102872us
I20260806 04:01:55.266099 5670 raft_consensus.cc:2726] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:01:55.319567 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 49820us
I20260806 04:01:55.325059 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:55.326850 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:55.333243 1137 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.334478 32475 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 04:01:55.267674 5442 consensus_queue.cc:1046] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260806 04:01:55.331561 5380 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:55.325771 5673 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:55.343426 32427 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.345149 32475 raft_consensus.cc:1261] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 04:01:55.301340 5422 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 11 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:55.346311 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:55.272238 5648 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:55.349612 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 82950.7us
I20260806 04:01:55.334195 5633 raft_consensus.cc:2741] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 04:01:55.328668 5470 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.351398 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:55.356107 32551 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.364003 32551 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.361714 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:55.354297 5163 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:55.347657 5221 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:55.345389 5337 raft_consensus.cc:2781] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:55.343159 5674 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:55.367004 32553 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.358090 5445 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.346539 5581 consensus_queue.cc:237] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:55.377843 5666 log.cc:924] Time spent T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260806 04:01:55.335655 5600 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:55.380443 5640 log.cc:924] Time spent T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.522s user 0.001s sys 0.001s
I20260806 04:01:55.384979 5673 raft_consensus.cc:2726] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:55.387511 32551 leader_election.cc:302] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:55.342105 5486 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:55.434515 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 47068.1us
I20260806 04:01:55.363819 5442 raft_consensus.cc:2781] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:55.349817 5008 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.039s
I20260806 04:01:55.383881 4849 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.448581 32466 raft_consensus.cc:1226] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:55.454710 32602 raft_consensus.cc:2396] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260806 04:01:55.388247 5337 raft_consensus.cc:686] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:55.367225 5580 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260806 04:01:55.358695 4679 consensus_queue.cc:1046] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
W20260806 04:01:55.457983 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 106963us
I20260806 04:01:55.437281 5444 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:55.438719 5442 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:55.444175 5674 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:55.476414 1184 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:55.388535 5221 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.371820 5682 consensus_queue.cc:1035] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260806 04:01:55.470973 4789 raft_consensus.cc:2726] T 9befcec6c17046459b03fa9ce98de94e P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:01:55.476063 5442 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:55.492669 5486 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:55.501294 5520 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.089s user 0.014s sys 0.018s
I20260806 04:01:55.501662 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2060) took 1004 ms. Trace:
I20260806 04:01:55.490631 5444 consensus_queue.cc:1035] T 45b7481674ca403b9a224047a6d99b03 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260806 04:01:55.502068 32615 rpcz_store.cc:276] 0806 04:01:54.497277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.625027 (+127750us) service_pool.cc:227] Handling call
0806 04:01:54.625317 (+ 290us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:55.170206 (+544889us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:55.501427 (+331221us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b5400216e54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:55.501599 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1251791232}
I20260806 04:01:55.502902 5648 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:55.478165 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 83633.7us
I20260806 04:01:55.509032 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:55.509677 32741 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:55.510227 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:55.511731 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:55.501693 5520 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.195s user 0.014s sys 0.018s
W20260806 04:01:55.523389 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 315071us
W20260806 04:01:55.514219 5488 consensus_meta.cc:287] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.552s user 0.008s sys 0.003s
I20260806 04:01:55.504209 5470 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:55.521551 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:55.527951 32741 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:55.529459 1189 raft_consensus.cc:1261] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260806 04:01:55.529844 1189 pending_rounds.cc:76] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Aborting all ops after (but not including) 6
I20260806 04:01:55.530035 1189 pending_rounds.cc:97] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 04:01:55.530241 1189 raft_consensus.cc:2866] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:55.528712 32737 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:55.469844 5422 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:55.538117 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 102040us
I20260806 04:01:55.541733 32551 leader_election.cc:302] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.544224 32682 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:55.545389 5463 raft_consensus.cc:2781] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:01:55.545900 5463 raft_consensus.cc:686] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [term 10 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:55.546692 5463 consensus_queue.cc:237] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.543491 5673 consensus_queue.cc:1035] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260806 04:01:55.551668 32748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33978 (request call id 1897) took 1102 ms. Trace:
I20260806 04:01:55.497609 5337 consensus_queue.cc:237] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.552070 32748 rpcz_store.cc:276] 0806 04:01:54.470608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.490871 (+ 20263us) service_pool.cc:227] Handling call
0806 04:01:54.491228 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:54.540203 (+ 48975us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:54.540232 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:54.862978 (+322746us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b48000d0758. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1720fb7f1 00007fd1720f28e7 00007fd1720f277a 00007fd1721e695f 00007fd1721e6627 00007fd1721fab2c 00007fd1721faad8 00007fd1721faa58 00007fd1721faa13 00007fd1721f9cbc 00007fd17117bbe6
0806 04:01:55.278630 (+415652us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b2c00108868. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:55.279137 (+ 507us) log.cc:842] Serialized 26 byte log entry
0806 04:01:55.351419 (+ 72282us) raft_consensus.cc:1631] Marking committed up to 3
0806 04:01:55.351458 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:55.351527 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:55.551417 (+199890us) raft_consensus.cc:1671] finished
0806 04:01:55.551439 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:55.551608 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48527,"raft.queue_time_us":51560,"raft.run_cpu_time_us":1532,"raft.run_wall_time_us":312818,"spinlock_wait_cycles":24502912,"thread_start_us":67705,"threads_started":1,"wal-append.queue_time_us":185360}
I20260806 04:01:55.554710 5268 raft_consensus.cc:3037] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:55.556197 5442 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.536667 5544 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.559156 1136 leader_election.cc:302] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
W20260806 04:01:55.576256 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 69136.9us
W20260806 04:01:55.619683 32605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42618 (request call id 766) took 3364 ms (client timeout 1651 ms). Trace:
W20260806 04:01:55.620133 32605 rpcz_store.cc:269] 0806 04:01:52.195873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:52.196781 (+ 908us) service_pool.cc:227] Handling call
0806 04:01:55.559870 (+3363089us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54002d0354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:55.560000 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3178253056}
I20260806 04:01:55.623608 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:55.542675 5713 raft_consensus.cc:2781] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:55.650987 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:55.633273 32469 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:01:55.650354 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:55.664063 1137 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters:
I20260806 04:01:55.659689 5612 raft_consensus.cc:2726] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:55.667284 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.658407 5221 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:55.653784 5713 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:55.619321 5442 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.683967 5612 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:55.684618 5612 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.686769 5612 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:01:55.697808 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 138733us
I20260806 04:01:55.671888 5584 raft_consensus.cc:2781] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:55.700835 32682 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:55.702404 5008 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:55.702950 5610 consensus_queue.cc:1046] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:55.704165 5610 raft_consensus.cc:3032] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:55.676555 5713 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:55.673272 5470 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:55.704581 5610 raft_consensus.cc:729] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:55.665802 5606 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.709542 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:55.711426 1136 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:55.713991 5606 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.716945 5481 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:55.700958 5584 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:55.718881 5584 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:55.706802 5008 raft_consensus.cc:686] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:55.720969 5008 consensus_queue.cc:237] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.722975 32477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2144) took 1116 ms. Trace:
I20260806 04:01:55.723361 32477 rpcz_store.cc:276] 0806 04:01:54.606480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.606876 (+ 396us) service_pool.cc:227] Handling call
0806 04:01:54.663049 (+ 56173us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:54.751935 (+ 88886us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:55.722685 (+970750us) spinlock_profiling.cc:243] Waited 75.6 ms on lock 0x7b540026fa54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:55.722905 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":173924224}
I20260806 04:01:55.726877 5551 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.730499 32474 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 8. Current term is 9. Ops: []
W20260806 04:01:55.732661 5538 log.cc:924] Time spent T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 2.437s user 0.004s sys 0.001s
W20260806 04:01:55.733026 5627 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.513s user 0.010s sys 0.000s
I20260806 04:01:55.733453 5627 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.734931 4784 raft_consensus.cc:3032] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:55.735342 4784 raft_consensus.cc:729] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:55.738180 5589 raft_consensus.cc:3032] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:55.738536 5589 raft_consensus.cc:729] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:55.739504 5268 raft_consensus.cc:2726] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:01:55.739277 5589 consensus_queue.cc:260] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.742970 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.743712 32605 raft_consensus.cc:2379] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 8: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:55.744048 32603 raft_consensus.cc:1226] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8. Ops: []
W20260806 04:01:55.744618 5669 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.535s user 0.003s sys 0.000s
I20260806 04:01:55.744848 5221 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:55.745172 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.745309 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.745558 5135 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:55.745877 32610 raft_consensus.cc:2354] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8.
I20260806 04:01:55.745963 32615 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 4.
I20260806 04:01:55.746716 5135 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.745595 5650 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:55.748059 5650 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.750975 5135 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:55.752336 4784 consensus_queue.cc:260] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.752594 5610 consensus_queue.cc:260] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.753623 4784 raft_consensus.cc:3037] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:55.753752 5610 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
W20260806 04:01:55.754382 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 46952.4us
I20260806 04:01:55.757215 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:55.757824 32748 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:55.758823 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:55.751827 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 970) took 1606 ms. Trace:
I20260806 04:01:55.760443 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:55.761090 5589 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:55.760219 1190 rpcz_store.cc:276] 0806 04:01:54.145510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.217342 (+ 71832us) service_pool.cc:227] Handling call
0806 04:01:55.751658 (+1534316us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b5400492ad4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:55.751759 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1202028288}
I20260806 04:01:55.765261 32738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33182 (request call id 706) took 1067 ms. Trace:
I20260806 04:01:55.765601 32738 rpcz_store.cc:276] 0806 04:01:54.697934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.698068 (+ 134us) service_pool.cc:227] Handling call
0806 04:01:54.744382 (+ 46314us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:54.761151 (+ 16769us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 04:01:54.868622 (+107471us) log.cc:842] Serialized 14 byte log entry
0806 04:01:55.446800 (+578178us) log.cc:842] Serialized 14 byte log entry
0806 04:01:55.464968 (+ 18168us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:55.480606 (+ 15638us) log.cc:842] Serialized 26 byte log entry
0806 04:01:55.481352 (+ 746us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:55.481513 (+ 161us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:55.481837 (+ 324us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:55.744119 (+262282us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:55.765074 (+ 20955us) raft_consensus.cc:1671] finished
0806 04:01:55.765095 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:55.765205 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34168,"prepare.queue_time_us":236754,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":44541,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":13343,"spinlock_wait_cycles":8583040,"thread_start_us":478209,"threads_started":2,"wal-append.queue_time_us":240441}
I20260806 04:01:55.766453 5430 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.768537 32748 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:55.769407 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.771477 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.770637 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.772452 32615 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:55.751449 5380 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.775496 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.806327 32612 raft_consensus.cc:2379] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 9: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:55.814819 32748 raft_consensus.cc:1226] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260806 04:01:55.830313 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.831216 5650 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.831668 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.832006 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.832795 32612 raft_consensus.cc:2379] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 11: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:55.833545 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.834616 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.835126 32612 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:55.837062 32553 leader_election.cc:302] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:55.838665 5470 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:55.839184 5470 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.841111 5470 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:55.841926 5650 raft_consensus.cc:2781] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 04:01:55.842183 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 142380us
I20260806 04:01:55.843397 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.844520 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.845557 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.846626 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.846721 1189 raft_consensus.cc:1261] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:55.848631 5488 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:55.849157 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.852424 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:55.867378 32608 raft_consensus.cc:1261] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260806 04:01:55.861050 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0201a31e3c594fdfa1cec8a9970becef" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:55.854162 1136 leader_election.cc:302] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:55.850879 1183 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 9 LEARNER]: Advancing to term 10
I20260806 04:01:55.870036 32608 pending_rounds.cc:76] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 5
I20260806 04:01:55.858723 1187 raft_consensus.cc:1261] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 04:01:55.871557 32608 pending_rounds.cc:97] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260806 04:01:55.871810 32608 raft_consensus.cc:2866] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:55.854005 1185 raft_consensus.cc:1261] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 04:01:55.857960 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:55.863776 32602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2049) took 1412 ms. Trace:
I20260806 04:01:55.874225 32602 rpcz_store.cc:276] 0806 04:01:54.451804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.453364 (+ 1560us) service_pool.cc:227] Handling call
0806 04:01:55.863584 (+1410220us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b5400086954. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:55.863718 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2137421184}
I20260806 04:01:55.869894 32604 raft_consensus.cc:2454] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:55.873705 32605 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:55.878185 1136 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:55.882383 32603 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 11.
I20260806 04:01:55.883708 32603 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 04:01:55.849542 5508 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:55.851161 5275 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:55.885170 5508 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.885280 5275 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.852730 5670 raft_consensus.cc:3032] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:55.849192 5488 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.887759 5670 raft_consensus.cc:729] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:55.745342 5221 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.888669 5670 consensus_queue.cc:260] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.889273 5488 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:55.889778 5670 raft_consensus.cc:3037] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:55.839190 5584 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:55.892513 1186 raft_consensus.cc:1226] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Rejecting Update request from peer dd7344d85a154539b7f8c484f2a76772 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:55.892823 4784 raft_consensus.cc:2726] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:55.894351 5584 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:55.894943 5584 consensus_queue.cc:1035] T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:55.897485 1185 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:55.898761 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:55.899091 5650 raft_consensus.cc:686] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:55.887291 5551 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:55.903272 32748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:55.904356 1189 raft_consensus.cc:3037] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:55.895176 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59498 (request call id 1825) took 1231 ms. Trace:
I20260806 04:01:55.905083 32614 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 7.
I20260806 04:01:55.892669 32604 raft_consensus.cc:1226] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:55.907791 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2048) took 1456 ms. Trace:
I20260806 04:01:55.908176 32609 rpcz_store.cc:276] 0806 04:01:54.451518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.451623 (+ 105us) service_pool.cc:227] Handling call
0806 04:01:55.907599 (+1455976us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b540007e4d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:55.907716 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44184320}
I20260806 04:01:55.909125 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:55.909590 32466 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 9. Current term is 10. Ops: []
I20260806 04:01:55.905380 32473 rpcz_store.cc:276] 0806 04:01:54.663765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.670157 (+ 6392us) service_pool.cc:227] Handling call
0806 04:01:55.894973 (+1224816us) spinlock_profiling.cc:243] Waited 86.3 ms on lock 0x7b5400201cd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:55.895093 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198379264}
I20260806 04:01:55.910985 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:55.911633 1186 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:55.912683 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:55.913398 1192 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 4.
I20260806 04:01:55.900331 5650 consensus_queue.cc:237] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:55.916556 32682 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:55.925153 32612 raft_consensus.cc:2454] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:55.928139 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.824260 5627 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:55.894357 5612 raft_consensus.cc:2781] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:55.923347 32551 leader_election.cc:302] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
W20260806 04:01:55.993507 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 76988.3us
I20260806 04:01:55.995080 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:56.001812 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2611a58e8424bd088918c8aa63add87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:56.002416 1192 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:55.944558 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:56.003226 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.003690 1186 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 11.
I20260806 04:01:55.941075 32602 raft_consensus.cc:1226] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: [4.6-5.7]
I20260806 04:01:55.999521 1189 raft_consensus.cc:2454] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 10.
I20260806 04:01:55.936612 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:55.940686 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:56.007531 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:55.938359 1183 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 10 LEARNER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 04:01:55.889797 5221 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:55.887343 5275 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:56.010912 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 87652.8us
I20260806 04:01:56.013324 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:56.014372 32612 raft_consensus.cc:2396] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 04:01:55.912181 5508 leader_election.cc:288] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.025276 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:56.026125 1184 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 10 LEARNER]: Advancing to term 11
I20260806 04:01:56.027758 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:56.029630 32602 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:56.033003 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:56.035235 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:56.000453 5463 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:56.038177 32604 raft_consensus.cc:1226] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260806 04:01:56.036516 32603 raft_consensus.cc:2454] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 10.
I20260806 04:01:56.041540 32551 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:56.054886 1189 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:56.015383 5606 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:56.017279 5470 raft_consensus.cc:2726] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:56.012895 5486 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:56.061697 32551 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:56.062803 32551 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:56.065954 1184 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 11 LEARNER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 11.
I20260806 04:01:55.993197 5618 consensus_queue.cc:1035] T e870f9de1d6a4db4bbf149c6a785135c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.101s
I20260806 04:01:56.071641 32682 leader_election.cc:398] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:56.073405 1136 leader_election.cc:302] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:56.085649 32682 leader_election.cc:401] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 04:01:56.086997 32604 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 10. Current term is 11. Ops: []
I20260806 04:01:56.037351 5612 raft_consensus.cc:686] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:56.012689 5589 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260806 04:01:55.994109 5636 consensus_queue.cc:1035] T a12b9e678e6049be86bca4e981f90379 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.063s
I20260806 04:01:56.094666 32609 raft_consensus.cc:1226] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:56.095914 1136 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:56.099748 1136 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:56.078689 5581 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.010s sys 0.012s
W20260806 04:01:56.081614 5596 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.210s user 0.002s sys 0.022s
W20260806 04:01:56.088657 32551 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.237.132:36181 timed out after 4.123s (SENT) [suppressed 24 similar messages]
I20260806 04:01:56.078776 5501 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:56.079077 5442 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:56.114943 32604 raft_consensus.cc:1261] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:01:56.092952 5613 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 6.
I20260806 04:01:56.091807 5445 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260806 04:01:56.103778 5581 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s user 0.010s sys 0.012s
W20260806 04:01:56.103952 5596 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.238s user 0.003s sys 0.023s
I20260806 04:01:56.133054 32748 raft_consensus.cc:3037] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:56.105832 5488 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:56.142091 5445 raft_consensus.cc:482] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:56.142688 5445 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Advancing to term 12
I20260806 04:01:56.144908 1192 raft_consensus.cc:1261] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260806 04:01:56.148653 5567 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 1.869s user 0.000s sys 0.004s
I20260806 04:01:56.150293 5380 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.113340 5442 raft_consensus.cc:482] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:56.156754 5442 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:56.109035 5469 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:56.078871 5486 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.151796 32682 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:56.058311 5627 consensus_queue.cc:1046] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
W20260806 04:01:56.105137 5341 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.493s user 0.019s sys 0.041s
I20260806 04:01:56.161936 5613 consensus_queue.cc:1035] T c4f865676f884755987b8bd820489186 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
W20260806 04:01:56.168609 5341 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.556s user 0.019s sys 0.041s
I20260806 04:01:56.168926 5627 raft_consensus.cc:3032] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:56.169104 5341 raft_consensus.cc:686] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:56.170193 5341 consensus_queue.cc:237] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.058966 4523 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260806 04:01:56.169389 5627 raft_consensus.cc:729] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:56.171828 5627 consensus_queue.cc:260] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.173102 5627 raft_consensus.cc:3037] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:56.159315 5469 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:56.174086 5469 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:56.174192 5520 consensus_queue.cc:1046] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:56.174938 5520 consensus_queue.cc:1225] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer f7c202b562ce482db283463d7f02e136 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 04:01:56.175720 5520 raft_consensus.cc:3032] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:56.176991 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:56.069439 5713 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.160054 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.165745 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.165185 1192 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260806 04:01:56.159685 32609 raft_consensus.cc:1261] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 04:01:56.159329 32748 raft_consensus.cc:1261] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 04:01:56.170596 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80edf8f3dcd4b64a167796d0c741ecd" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:56.113314 5501 raft_consensus.cc:482] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:56.131376 5612 consensus_queue.cc:237] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.174348 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2062) took 1676 ms. Trace:
I20260806 04:01:56.178889 32741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:56.188036 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.181329 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.188467 32608 rpcz_store.cc:276] 0806 04:01:54.497796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.635839 (+138043us) service_pool.cc:227] Handling call
0806 04:01:55.867019 (+1231180us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:55.872207 (+ 5188us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.174156 (+301949us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400086950. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:56.174284 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2766880512}
I20260806 04:01:56.162456 32682 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:56.162416 5610 consensus_queue.cc:1035] T 187160a8bf734a23b90fcc49bc24b10b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20260806 04:01:56.163780 4922 consensus_queue.cc:1046] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.029s
I20260806 04:01:56.192891 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.161661 5511 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:56.193688 32474 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:56.194491 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:56.195109 32474 raft_consensus.cc:3037] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:56.203631 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1948a6fbe64767a22598e910340a6b" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.208844 32615 raft_consensus.cc:2379] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 in current term 9: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:56.210071 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.209137 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a23c953ef64542931fb8ede0fcbb97" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:56.211365 1189 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:56.212451 32469 raft_consensus.cc:2379] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 5: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:56.161103 5486 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.167951 5273 consensus_queue.cc:1046] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260806 04:01:56.210712 32475 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:56.163517 4679 raft_consensus.cc:3032] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:56.237118 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.237713 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:56.237476 32682 leader_election.cc:302] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:56.194872 5501 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:56.248116 32608 raft_consensus.cc:1226] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:56.241904 32604 raft_consensus.cc:1226] T 0bf6a30af60042f9b0a84eb0fdb8db19 P dd7344d85a154539b7f8c484f2a76772 [term 7 LEADER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:56.176043 5520 raft_consensus.cc:729] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:56.244673 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:56.250308 32470 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:56.287114 1184 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260806 04:01:56.173311 5485 consensus_queue.cc:1046] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
W20260806 04:01:56.295572 32610 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.043s sys 0.040s
I20260806 04:01:56.296216 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62779f9120ed4ef3b0d9febc7c2cd224" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:56.299494 32610 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s user 0.043s sys 0.040s
I20260806 04:01:56.305150 1189 raft_consensus.cc:2454] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 10.
I20260806 04:01:56.203004 5511 raft_consensus.cc:686] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:56.209235 5610 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:56.313349 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:56.324338 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 127308us
I20260806 04:01:56.201455 5275 consensus_queue.cc:1035] T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.106s
W20260806 04:01:56.326259 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 88869.3us
I20260806 04:01:56.327349 32427 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.240940 4679 raft_consensus.cc:729] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:56.323843 5684 log.cc:924] Time spent T 62779f9120ed4ef3b0d9febc7c2cd224 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
W20260806 04:01:56.309832 5661 log.cc:924] Time spent T 0bf6a30af60042f9b0a84eb0fdb8db19 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260806 04:01:56.331197 5252 raft_consensus.cc:2726] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:56.318948 5584 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.202847 5445 raft_consensus.cc:504] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.338902 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.347506 32475 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 6.
I20260806 04:01:56.354496 32427 leader_election.cc:398] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Vote denied by peer dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857) with higher term. Message: Invalid argument: T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8.
I20260806 04:01:56.355392 32427 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.353437 1137 leader_election.cc:302] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.246310 5594 consensus_queue.cc:1046] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.095s
I20260806 04:01:56.366534 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
W20260806 04:01:56.376582 32467 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.024s sys 0.066s
W20260806 04:01:56.376971 32467 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.024s sys 0.066s
I20260806 04:01:56.384721 32553 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.387207 32553 leader_election.cc:302] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:56.415606 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 60292.5us
I20260806 04:01:56.416452 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
W20260806 04:01:56.417768 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 52091.7us
I20260806 04:01:56.418871 32682 leader_election.cc:302] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:56.361279 5610 raft_consensus.cc:686] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:56.366935 5546 consensus_queue.cc:1046] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260806 04:01:56.360611 5584 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.411090 4637 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:56.353684 4784 raft_consensus.cc:2726] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:56.433980 32745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33964 (request call id 819) took 1488 ms. Trace:
I20260806 04:01:56.452529 32745 rpcz_store.cc:276] 0806 04:01:54.945704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.988235 (+ 42531us) service_pool.cc:227] Handling call
0806 04:01:56.433271 (+1445036us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54000a87d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9af0 00007fd16d25cf2a 00007fd172199c6e 00007fd16b890807 00007fd16b890792 00007fd16b8906c1 00007fd16b89063c 00007fd16b88f8f2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:56.433912 (+ 641us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1123227008}
I20260806 04:01:56.423358 32739 raft_consensus.cc:1226] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:56.421607 32427 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.411564 5008 consensus_queue.cc:1035] T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260806 04:01:56.443686 5136 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.015s
I20260806 04:01:56.441627 5032 raft_consensus.cc:2726] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 04:01:56.460891 5546 raft_consensus.cc:3032] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:56.460666 5610 consensus_queue.cc:237] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.461318 5546 raft_consensus.cc:729] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:56.462150 5546 consensus_queue.cc:260] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.463440 5546 raft_consensus.cc:3037] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:56.440459 5713 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.464537 5670 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.433197 5511 consensus_queue.cc:237] T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.465087 5670 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.421427 4922 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260806 04:01:56.468155 5670 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.442864 5613 consensus_queue.cc:1035] T ec440a6dfdfc46af8a0876b464ef531d P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260806 04:01:56.403895 4523 raft_consensus.cc:2726] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:56.470674 1184 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260806 04:01:56.471567 5732 log.cc:924] Time spent T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260806 04:01:56.472884 5633 consensus_queue.cc:1046] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:56.473634 5633 consensus_queue.cc:1225] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer f7c202b562ce482db283463d7f02e136 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 04:01:56.376732 4849 raft_consensus.cc:2726] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:56.421072 5273 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.048s
W20260806 04:01:56.481874 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 60387.3us
I20260806 04:01:56.486881 32427 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.404119 5486 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:56.437302 4679 consensus_queue.cc:260] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.488489 4679 raft_consensus.cc:3037] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:56.414633 5541 raft_consensus.cc:2781] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:56.492304 5541 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:56.492736 5541 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:56.494239 5600 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.495831 32466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2164) took 1163 ms. Trace:
I20260806 04:01:56.496266 32466 rpcz_store.cc:276] 0806 04:01:55.332049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:55.456700 (+124651us) service_pool.cc:227] Handling call
0806 04:01:55.457030 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:55.909961 (+452931us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.495557 (+585596us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b54000497d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:56.495754 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1040398720}
I20260806 04:01:56.497793 32466 raft_consensus.cc:1226] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:56.424670 5737 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:56.500545 32605 raft_consensus.cc:1226] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:56.501693 5445 leader_election.cc:288] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 12 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.430454 5495 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:56.502386 5495 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:56.502737 5495 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:56.504361 5430 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.506801 5521 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.507721 5527 raft_consensus.cc:3032] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:56.436574 5551 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:56.509186 5584 leader_election.cc:288] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.507359 5521 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.491957 5485 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.512856 5527 raft_consensus.cc:729] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:56.529719 32427 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.534206 32470 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 11.
I20260806 04:01:56.399711 32553 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:56.540112 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 140477us
I20260806 04:01:56.564682 32553 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.514029 5551 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:56.565388 32474 raft_consensus.cc:2454] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:56.516613 5430 raft_consensus.cc:2726] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 04:01:56.572526 32427 leader_election.cc:302] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.537840 5484 raft_consensus.cc:2726] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:01:56.540679 5548 raft_consensus.cc:2726] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:56.540484 5674 raft_consensus.cc:2726] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 04:01:56.511878 5520 consensus_queue.cc:260] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.562039 4849 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.516237 5485 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.658339 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.658602 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.659273 32749 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:56.663720 32428 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:56.668510 1187 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.028s sys 0.045s
W20260806 04:01:56.668898 1187 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s user 0.028s sys 0.045s
I20260806 04:01:56.669212 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:56.601440 32475 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260806 04:01:56.559672 4993 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.576797 5600 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:56.570451 5551 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:56.613646 32474 raft_consensus.cc:1226] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:56.628538 5520 raft_consensus.cc:3037] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:56.596130 5380 consensus_queue.cc:1046] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260806 04:01:56.625092 5469 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.609385 5589 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260806 04:01:56.607686 5481 raft_consensus.cc:2781] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:56.556855 4922 consensus_queue.cc:1046] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260806 04:01:56.615219 5527 consensus_queue.cc:260] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.556368 5613 consensus_queue.cc:1046] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
W20260806 04:01:56.640043 5631 log.cc:924] Time spent T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 1.817s user 0.008s sys 0.012s
I20260806 04:01:56.500490 5737 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.628052 4637 raft_consensus.cc:2726] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:56.655215 5485 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.629204 5627 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.691391 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.652606 5521 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.668654 4679 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:54.767582 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 04:01:56.717324 32749 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33964
I20260806 04:01:56.731720 32749 raft_consensus.cc:595] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 LEADER]: Received request to transfer leadership
I20260806 04:01:56.716398 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.628221 4849 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.720624 5032 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.734216 4679 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:56.628793 5495 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.640581 5541 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.746225 32478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2143) took 2185 ms. Trace:
I20260806 04:01:56.746592 32478 rpcz_store.cc:276] 0806 04:01:54.561003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.561317 (+ 314us) service_pool.cc:227] Handling call
0806 04:01:54.606641 (+ 45324us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:54.780157 (+173516us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.746009 (+1965852us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b540019a354. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:56.746170 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378721536}
W20260806 04:01:56.747404 32471 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.015s sys 0.059s
W20260806 04:01:56.747771 32471 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s user 0.015s sys 0.059s
I20260806 04:01:56.734400 5032 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.750484 5469 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.731823 5527 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:56.752287 5620 consensus_queue.cc:1046] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:56.742470 5627 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.753158 5620 raft_consensus.cc:3032] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:56.753522 5620 raft_consensus.cc:729] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:56.754463 5627 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.754292 5620 consensus_queue.cc:260] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.755510 5620 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:56.722277 5481 raft_consensus.cc:686] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:56.757879 32745 raft_consensus.cc:1261] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:01:56.759052 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.684986 4993 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.721359 5273 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:56.768529 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
W20260806 04:01:56.769080 1186 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.024s sys 0.055s
I20260806 04:01:56.756260 4849 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.771339 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.737099 4679 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:01:56.777129 1186 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s user 0.024s sys 0.056s
I20260806 04:01:56.735405 5501 raft_consensus.cc:504] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.776141 32553 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
W20260806 04:01:56.781642 1185 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.033s sys 0.046s
W20260806 04:01:56.790002 1185 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s user 0.033s sys 0.046s
I20260806 04:01:56.789804 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
W20260806 04:01:56.784919 32551 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.005s sys 0.017s
W20260806 04:01:56.778083 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 397515us
W20260806 04:01:56.791478 32551 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.681s user 0.005s sys 0.017s
W20260806 04:01:56.792866 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 704180us
I20260806 04:01:56.772567 5627 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.769269 5032 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.811296 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f865676f884755987b8bd820489186" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:56.813503 32739 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:56.809899 5541 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.815217 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:56.814905 5481 consensus_queue.cc:237] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.812765 5633 raft_consensus.cc:482] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:56.820390 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:56.820585 5633 raft_consensus.cc:504] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.822575 5633 leader_election.cc:288] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.775074 5273 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.824265 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.825067 32469 raft_consensus.cc:2379] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 7: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:56.825382 5273 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.825824 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:56.826974 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.827262 32553 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.830705 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:56.831364 32609 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 8.
I20260806 04:01:56.832401 32466 raft_consensus.cc:1261] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 04:01:56.821012 32603 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:56.837309 5674 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:56.775162 4523 consensus_queue.cc:1035] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260806 04:01:56.837895 5674 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.838642 4993 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:56.844061 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.845795 32551 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.846082 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.853911 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:56.854188 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:56.781934 5682 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.006s sys 0.010s
I20260806 04:01:56.854862 32478 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
I20260806 04:01:56.857867 32603 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:56.858347 32615 raft_consensus.cc:2454] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:56.854718 1183 raft_consensus.cc:2454] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:56.861505 5032 consensus_queue.cc:1035] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:56.861728 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:56.862913 5713 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Leader election won for term 11
I20260806 04:01:56.838673 4523 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260806 04:01:56.863803 5442 consensus_meta.cc:287] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.707s user 0.011s sys 0.000s
I20260806 04:01:56.864230 5442 raft_consensus.cc:504] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.864704 32609 raft_consensus.cc:1226] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: []
I20260806 04:01:56.865988 32609 raft_consensus.cc:1226] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:56.864252 5380 consensus_queue.cc:1046] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260806 04:01:56.866550 5589 consensus_queue.cc:1046] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 04:01:56.866379 5713 raft_consensus.cc:686] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [term 11 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:56.868481 5713 consensus_queue.cc:237] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.870002 5633 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:56.870510 5633 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:56.872136 5495 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.852205 5486 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.872967 5486 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.874836 5486 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.841073 5674 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.870018 5501 leader_election.cc:288] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.863191 32614 raft_consensus.cc:1261] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 04:01:56.876770 5273 consensus_queue.cc:1046] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:01:56.877832 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:56.878412 5273 raft_consensus.cc:3032] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 LEADER]: Stepping down as leader of term 10
I20260806 04:01:56.876718 5737 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.878818 5273 raft_consensus.cc:729] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:56.872434 5633 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:56.878428 32466 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:56.815717 5627 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.881266 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:56.881699 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.875587 5551 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.882591 5627 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.810779 5445 raft_consensus.cc:2726] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:56.884227 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.884567 5551 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.885857 1136 leader_election.cc:302] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:56.887320 1136 leader_election.cc:302] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:56.888140 5544 raft_consensus.cc:2781] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:56.889024 1136 leader_election.cc:302] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:56.863147 32608 raft_consensus.cc:1261] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260806 04:01:56.894021 1136 leader_election.cc:302] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:56.889942 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.876709 32478 raft_consensus.cc:1226] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:56.894134 32602 raft_consensus.cc:1261] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260806 04:01:56.893743 32608 pending_rounds.cc:76] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Aborting all ops after (but not including) 6
I20260806 04:01:56.894924 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:56.897800 32608 pending_rounds.cc:97] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260806 04:01:56.899314 32608 raft_consensus.cc:2866] T c2611a58e8424bd088918c8aa63add87 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:56.898324 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.901654 32682 leader_election.cc:302] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:56.859776 5682 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.624s user 0.006s sys 0.011s
I20260806 04:01:56.905779 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e870f9de1d6a4db4bbf149c6a785135c" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:56.906562 5589 consensus_queue.cc:1035] T db65aabb33f8427593eac56e96ca9cc6 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260806 04:01:56.909809 5737 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:56.914547 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.922358 32466 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:56.866204 5442 leader_election.cc:288] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:56.925199 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:56.926048 32469 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 6: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:56.929100 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7172fff185b4788acb609d2314e4b4b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:56.929189 32473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.000s sys 0.044s
I20260806 04:01:56.927187 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.930881 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:56.894279 5544 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:56.936269 32745 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 5.
I20260806 04:01:56.934203 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:56.933655 32553 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters:
W20260806 04:01:56.935783 32473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.000s sys 0.044s
I20260806 04:01:56.900156 5337 raft_consensus.cc:2781] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:56.942379 5442 raft_consensus.cc:2781] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 04:01:56.942873 5442 raft_consensus.cc:686] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [term 8 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:56.944222 32466 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:56.943756 5442 consensus_queue.cc:237] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:56.914628 5612 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.013s sys 0.010s
W20260806 04:01:56.928267 5252 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.015s sys 0.030s
W20260806 04:01:56.935664 5341 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.014s sys 0.011s
I20260806 04:01:56.947435 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 12 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:56.923326 5495 raft_consensus.cc:2781] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 04:01:56.898092 5519 consensus_queue.cc:1035] T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260806 04:01:56.928246 4993 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:56.952710 1190 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.014s sys 0.016s
W20260806 04:01:56.956774 1190 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s user 0.014s sys 0.016s
I20260806 04:01:56.934988 5627 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.879719 5273 consensus_queue.cc:260] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:56.945533 5544 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:01:56.948632 5252 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.015s sys 0.031s
I20260806 04:01:56.945263 32469 raft_consensus.cc:1226] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 8. Current term is 9. Ops: []
W20260806 04:01:56.950587 5612 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.013s sys 0.010s
W20260806 04:01:56.951256 5341 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.015s sys 0.012s
I20260806 04:01:56.935767 5008 consensus_queue.cc:1035] T 8417acc05db341aca15704cd05736651 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260806 04:01:56.931075 5584 consensus_queue.cc:1035] T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.063s
I20260806 04:01:56.994987 32609 raft_consensus.cc:1261] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Refusing update from remote peer d4a238f055a44f459c62c224daa4d541: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 04:01:57.012730 5478 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:57.016244 5610 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.009s sys 0.045s
I20260806 04:01:56.989085 4849 raft_consensus.cc:2726] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:57.022169 32469 raft_consensus.cc:1226] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:57.029749 32466 raft_consensus.cc:2454] T d2ad471021a947e9a45d57c7be704453 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
W20260806 04:01:57.024698 5610 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s user 0.009s sys 0.046s
I20260806 04:01:57.052207 5737 raft_consensus.cc:2726] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:57.015509 5613 consensus_queue.cc:1035] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
I20260806 04:01:57.053650 4679 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:57.014689 5470 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:56.991214 4993 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.006137 5627 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:57.066107 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 460347us
W20260806 04:01:57.094722 32610 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.016s sys 0.034s
W20260806 04:01:57.095342 32610 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.535s user 0.016s sys 0.035s
I20260806 04:01:57.141868 32610 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2232) took 1371 ms. Trace:
I20260806 04:01:57.142354 32610 rpcz_store.cc:276] 0806 04:01:55.770511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:55.770624 (+ 113us) service_pool.cc:227] Handling call
0806 04:01:55.770963 (+ 339us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:55.771181 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:55.771204 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:55.771642 (+ 438us) log.cc:842] Serialized 26 byte log entry
0806 04:01:56.299947 (+528305us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:57.095786 (+795839us) log.cc:842] Serialized 14 byte log entry
0806 04:01:57.141464 (+ 45678us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.141559 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.141616 (+ 57us) raft_consensus.cc:1671] finished
0806 04:01:57.141632 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.141790 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45313,"thread_start_us":1063121,"threads_started":2,"wal-append.queue_time_us":726738}
I20260806 04:01:57.036334 5478 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:57.142961 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 67951.9us
I20260806 04:01:57.156214 32427 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.158553 32427 leader_election.cc:302] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.145494 5674 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:57.178994 32745 raft_consensus.cc:1226] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:57.141721 5521 consensus_queue.cc:1035] T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260806 04:01:57.172295 5380 raft_consensus.cc:2726] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 04:01:57.082669 5470 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.209964 32604 raft_consensus.cc:2932] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:01:57.216331 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 287497us
I20260806 04:01:57.239176 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 1012) took 1385 ms. Trace:
W20260806 04:01:57.242811 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 84347.5us
I20260806 04:01:57.173635 5633 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
W20260806 04:01:57.248057 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 317998us
I20260806 04:01:57.239779 1187 rpcz_store.cc:276] 0806 04:01:55.853625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:55.899445 (+ 45820us) service_pool.cc:227] Handling call
0806 04:01:55.899779 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:55.900140 (+ 361us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:55.900166 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:55.900638 (+ 472us) log.cc:842] Serialized 26 byte log entry
0806 04:01:56.669266 (+768628us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:56.689858 (+ 20592us) log.cc:842] Serialized 14 byte log entry
0806 04:01:56.712324 (+ 22466us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.712425 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.238925 (+526500us) raft_consensus.cc:1671] finished
0806 04:01:57.238948 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.239101 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76524,"prepare.queue_time_us":20483,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":5144,"raft.queue_time_us":26818,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":35659,"spinlock_wait_cycles":7913856,"thread_start_us":768257,"threads_started":1,"wal-append.queue_time_us":1036741}
I20260806 04:01:57.251302 32609 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 12 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 11. Current term is 12. Ops: []
I20260806 04:01:57.254972 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:57.255831 32608 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:57.183597 5674 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.259497 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:57.236928 5546 raft_consensus.cc:2781] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 04:01:57.260910 32603 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:57.265669 32427 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541
I20260806 04:01:57.266512 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf5ed335f5bb4cf491d1d169c809b5d9" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:57.267923 4922 raft_consensus.cc:3032] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 04:01:57.268778 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2166) took 1022 ms. Trace:
I20260806 04:01:57.294888 32475 rpcz_store.cc:276] 0806 04:01:56.246443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.352073 (+105630us) service_pool.cc:227] Handling call
0806 04:01:56.559054 (+206981us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:56.674027 (+114973us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.268492 (+594465us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b5400012f54. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:57.268708 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96912768}
I20260806 04:01:57.270299 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 1014) took 1361 ms. Trace:
I20260806 04:01:57.271209 5470 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:57.276636 5770 log.cc:924] Time spent T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260806 04:01:57.309985 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:57.310647 32737 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
W20260806 04:01:57.277352 5778 log.cc:924] Time spent T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 04:01:57.236896 5541 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:57.312526 5541 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.279410 4679 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.280056 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:57.264341 5610 consensus_queue.cc:1046] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260806 04:01:57.282264 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:57.314445 1187 raft_consensus.cc:2454] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 7.
I20260806 04:01:57.315521 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fdfcc3345c4c86a7d974030c67c271" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:57.316134 1187 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Advancing to term 12
I20260806 04:01:57.285326 32610 raft_consensus.cc:1261] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260806 04:01:57.297879 32475 raft_consensus.cc:1261] T 9f7858566354435c9b6b09c5b8c1714c P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 04:01:57.302887 32603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:57.325205 32603 raft_consensus.cc:2454] T db65aabb33f8427593eac56e96ca9cc6 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:57.304211 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:01:57.326154 5756 log.cc:924] Time spent T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.261s user 0.000s sys 0.000s
I20260806 04:01:57.328378 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2067) took 1494 ms. Trace:
I20260806 04:01:57.328806 32612 rpcz_store.cc:276] 0806 04:01:55.833410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.019134 (+185724us) service_pool.cc:227] Handling call
0806 04:01:56.045698 (+ 26564us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:56.061370 (+ 15672us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 04:01:56.090144 (+ 28774us) log.cc:842] Serialized 14 byte log entry
0806 04:01:56.598061 (+507917us) log.cc:842] Serialized 14 byte log entry
0806 04:01:56.598187 (+ 126us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:56.613403 (+ 15216us) log.cc:842] Serialized 52 byte log entry
0806 04:01:56.613534 (+ 131us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:56.613564 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.613623 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.275972 (+662349us) spinlock_profiling.cc:243] Waited 4.81 ms on lock 0x7b18004b6c40. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a32c638 00007fd16a32fb11 00007fd16a32fac9 00007fd16a32fa59 00007fd16a32fa21 00007fd16a32ed1d 00007fd173590204 00007fd173590039 00007fd16a30edc6
0806 04:01:57.328126 (+ 52154us) raft_consensus.cc:1671] finished
0806 04:01:57.328151 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.328286 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15877,"prepare.queue_time_us":727214,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":9818,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":10317,"spinlock_wait_cycles":11054976,"thread_start_us":449394,"threads_started":1,"wal-append.queue_time_us":910709}
I20260806 04:01:57.328379 5541 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:57.332525 1187 raft_consensus.cc:2454] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 12.
I20260806 04:01:57.333581 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:57.334185 1187 raft_consensus.cc:3037] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:57.306310 5268 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:57.336371 5268 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:57.338451 5268 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.296003 1185 rpcz_store.cc:276] 0806 04:01:55.909109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:55.914676 (+ 5567us) service_pool.cc:227] Handling call
0806 04:01:55.993779 (+ 79103us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:56.020337 (+ 26558us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.020370 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:56.020845 (+ 475us) log.cc:842] Serialized 26 byte log entry
0806 04:01:56.790483 (+769638us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:56.859198 (+ 68715us) log.cc:842] Serialized 14 byte log entry
0806 04:01:56.883555 (+ 24357us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.883619 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.270042 (+386423us) raft_consensus.cc:1671] finished
0806 04:01:57.270071 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.270235 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50232,"prepare.queue_time_us":68251,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":60402,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":6312,"thread_start_us":731065,"threads_started":1,"wal-append.queue_time_us":1184759}
I20260806 04:01:57.306205 4922 raft_consensus.cc:729] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:57.339563 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b58a6c0e4e409fb675ad1e01ef8b60" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:57.339823 4922 consensus_queue.cc:260] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:57.340759 5749 log.cc:924] Time spent T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.274s user 0.000s sys 0.000s
I20260806 04:01:57.342164 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 1017) took 1432 ms. Trace:
I20260806 04:01:57.342547 1186 rpcz_store.cc:276] 0806 04:01:55.909972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.004298 (+ 94326us) service_pool.cc:227] Handling call
0806 04:01:56.004603 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:56.004814 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.004835 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:56.028478 (+ 23643us) log.cc:842] Serialized 26 byte log entry
0806 04:01:56.785943 (+757465us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:56.785984 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.786051 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.341950 (+555899us) raft_consensus.cc:1671] finished
0806 04:01:57.341973 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.342102 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":709994,"threads_started":1,"wal-append.queue_time_us":986220}
I20260806 04:01:57.344107 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:57.345214 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2118a2ac1a4e3b873d360b7ef9b47e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 12 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:57.313290 32745 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:57.351297 1186 raft_consensus.cc:2454] T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 11.
I20260806 04:01:57.312677 32614 raft_consensus.cc:1226] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 04:01:57.354365 1185 raft_consensus.cc:3037] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 8
W20260806 04:01:57.348793 5766 log.cc:924] Time spent T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.272s user 0.001s sys 0.001s
I20260806 04:01:57.354311 1187 raft_consensus.cc:2454] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 6.
I20260806 04:01:57.361377 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:57.386541 32474 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 9: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:57.361685 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:57.377022 32737 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
W20260806 04:01:57.324785 5758 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.264s user 0.000s sys 0.001s
I20260806 04:01:57.352504 5543 raft_consensus.cc:3032] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:57.382369 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:57.396098 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 136597us
I20260806 04:01:57.403255 1186 raft_consensus.cc:2454] T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
I20260806 04:01:57.404215 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6023572bff444c85842d7b3351debf87" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:57.405057 1186 raft_consensus.cc:2379] T 6023572bff444c85842d7b3351debf87 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 6: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:01:57.415280 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:57.415982 1186 raft_consensus.cc:2454] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
I20260806 04:01:57.416821 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:57.284581 5341 consensus_queue.cc:1046] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20260806 04:01:57.357311 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.424479 32739 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 5.
I20260806 04:01:57.309577 5674 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:57.420502 32551 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
W20260806 04:01:57.421121 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 155558us
I20260806 04:01:57.308146 5380 consensus_queue.cc:1046] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260806 04:01:57.426317 32551 leader_election.cc:302] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:57.429409 1185 raft_consensus.cc:2454] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 8.
I20260806 04:01:57.437975 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:57.440644 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20f5ce3f2ac40379306e3d630f63eda" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:57.442437 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 183213us
I20260806 04:01:57.435916 32551 leader_election.cc:302] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:57.403764 5627 leader_election.cc:288] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.430088 32428 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:57.431339 32739 raft_consensus.cc:3032] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:57.469923 32739 raft_consensus.cc:729] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:57.413841 4993 leader_election.cc:288] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.400528 5543 raft_consensus.cc:729] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:57.470964 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 43673.4us
I20260806 04:01:57.420269 5546 raft_consensus.cc:686] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:57.390363 4523 consensus_queue.cc:1035] T 9f7858566354435c9b6b09c5b8c1714c P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
W20260806 04:01:57.481750 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 45919.2us
I20260806 04:01:57.516636 32551 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:57.517944 32551 leader_election.cc:302] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.473279 5478 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.390753 5521 consensus_queue.cc:1046] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260806 04:01:57.484690 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:57.394162 5470 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:57.368927 5337 raft_consensus.cc:686] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:57.470794 32739 consensus_queue.cc:260] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:57.536784 32739 raft_consensus.cc:3037] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:57.427837 5765 consensus_queue.cc:1035] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.084s
I20260806 04:01:57.383481 5495 raft_consensus.cc:686] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:57.432520 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
W20260806 04:01:57.486330 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 56329.3us
I20260806 04:01:57.438274 5548 raft_consensus.cc:2741] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
W20260806 04:01:57.493623 5773 log.cc:924] Time spent T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
W20260806 04:01:57.494287 5764 log.cc:924] Time spent T d7172fff185b4788acb609d2314e4b4b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20260806 04:01:57.495134 5767 log.cc:924] Time spent T 305cbd07ce124048b7ac2f99b2aa40db P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
W20260806 04:01:57.495785 5775 log.cc:924] Time spent T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.258s user 0.000s sys 0.000s
I20260806 04:01:57.470588 5252 consensus_queue.cc:1035] T 7ec5d99c8efa405b84db49741c2f8933 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.026s
I20260806 04:01:57.496922 5801 consensus_queue.cc:1046] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:01:57.532514 5674 raft_consensus.cc:2781] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 04:01:57.527493 5473 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:57.498924 5341 raft_consensus.cc:3032] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:01:57.469549 5221 raft_consensus.cc:2781] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 04:01:57.436689 5544 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.501119 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 933) took 1874 ms. Trace:
I20260806 04:01:57.574651 5674 raft_consensus.cc:686] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [term 6 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:57.575096 32471 rpcz_store.cc:276] 0806 04:01:55.626928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:55.659139 (+ 32211us) service_pool.cc:227] Handling call
0806 04:01:55.668171 (+ 9032us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:56.176294 (+508123us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.176324 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:56.176773 (+ 449us) log.cc:842] Serialized 26 byte log entry
0806 04:01:56.767071 (+590298us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:56.767115 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.768028 (+ 913us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.500891 (+732863us) raft_consensus.cc:1671] finished
0806 04:01:57.500912 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.501060 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":555025,"raft.queue_time_us":516180,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"thread_start_us":570952,"threads_started":1,"wal-append.queue_time_us":816569}
I20260806 04:01:57.576015 5674 consensus_queue.cc:237] T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:57.577042 5786 log.cc:924] Time spent T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.152s user 0.000s sys 0.000s
I20260806 04:01:57.472007 5714 raft_consensus.cc:2781] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 12 FOLLOWER]: Leader election won for term 12
I20260806 04:01:57.577579 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 889) took 1051 ms. Trace:
W20260806 04:01:57.503084 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 132367us
I20260806 04:01:57.580044 5544 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.420568 5469 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:57.580833 5469 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.582851 5469 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.520004 32551 leader_election.cc:302] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:57.583936 1185 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 6.
I20260806 04:01:57.489859 5485 raft_consensus.cc:2781] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:57.517789 32427 leader_election.cc:302] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.584759 5485 raft_consensus.cc:686] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [term 9 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:57.585216 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 67505.3us
I20260806 04:01:57.585567 5546 consensus_queue.cc:237] T e5a23c953ef64542931fb8ede0fcbb97 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.585805 5485 consensus_queue.cc:237] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:01:57.532452 32749 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.004s sys 0.013s
I20260806 04:01:57.560384 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:57.561617 1137 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.010s sys 0.008s
W20260806 04:01:57.587410 1137 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.652s user 0.010s sys 0.008s
I20260806 04:01:57.574712 5341 raft_consensus.cc:729] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:57.589672 5341 consensus_queue.cc:260] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.590930 5341 raft_consensus.cc:3037] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:57.586853 5548 raft_consensus.cc:2726] T bf5ed335f5bb4cf491d1d169c809b5d9 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 04:01:57.497877 5736 log.cc:924] Time spent T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.846s user 0.000s sys 0.008s
I20260806 04:01:57.592192 5221 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:57.587122 32749 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s user 0.004s sys 0.013s
I20260806 04:01:57.577149 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 885) took 1133 ms. Trace:
I20260806 04:01:57.594350 5221 raft_consensus.cc:3037] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:57.488605 5627 raft_consensus.cc:2781] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260806 04:01:57.593385 1189 rpcz_store.cc:276] 0806 04:01:56.443596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.446023 (+ 2427us) service_pool.cc:227] Handling call
0806 04:01:56.446379 (+ 356us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:56.446728 (+ 349us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.446754 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:56.447256 (+ 502us) log.cc:842] Serialized 52 byte log entry
0806 04:01:56.937876 (+490620us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:56.952387 (+ 14511us) log.cc:842] Serialized 14 byte log entry
0806 04:01:57.393181 (+440794us) log.cc:842] Serialized 14 byte log entry
0806 04:01:57.393841 (+ 660us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.393924 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.576922 (+182998us) raft_consensus.cc:1671] finished
0806 04:01:57.576945 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.577091 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":215,"prepare.queue_time_us":15454,"prepare.run_cpu_time_us":2376,"prepare.run_wall_time_us":16883,"raft.queue_time_us":79709,"raft.run_cpu_time_us":548,"raft.run_wall_time_us":45982,"spinlock_wait_cycles":910080,"thread_start_us":930282,"threads_started":2,"wal-append.queue_time_us":727544}
I20260806 04:01:57.586009 32427 leader_election.cc:302] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.586853 5442 raft_consensus.cc:2726] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:57.588873 1186 raft_consensus.cc:1226] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 11 LEARNER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 10. Current term is 11. Ops: [9.9-10.10]
I20260806 04:01:57.531477 5713 raft_consensus.cc:2781] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:57.566107 32428 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:57.538290 5470 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.574759 5543 consensus_queue.cc:260] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.597309 1137 leader_election.cc:302] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:57.601609 32475 raft_consensus.cc:1261] T 11b58a6c0e4e409fb675ad1e01ef8b60 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 04:01:57.577998 1184 rpcz_store.cc:276] 0806 04:01:56.526400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.526537 (+ 137us) service_pool.cc:227] Handling call
0806 04:01:56.559958 (+ 33421us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:56.560334 (+ 376us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.560364 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:56.836528 (+276164us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b2c00127298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:56.836946 (+ 418us) log.cc:842] Serialized 26 byte log entry
0806 04:01:57.251874 (+414928us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:57.251936 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.252006 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.577252 (+325246us) raft_consensus.cc:1671] finished
0806 04:01:57.577270 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.577490 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1751,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":12356,"spinlock_wait_cycles":2717056,"thread_start_us":414462,"threads_started":1,"wal-append.queue_time_us":738113}
W20260806 04:01:57.610857 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 44826.5us
I20260806 04:01:57.608314 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 893) took 1045 ms. Trace:
I20260806 04:01:57.609687 32427 leader_election.cc:302] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.619738 1137 leader_election.cc:302] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:57.531420 4679 consensus_queue.cc:1046] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260806 04:01:57.558022 5612 raft_consensus.cc:482] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:57.571230 5423 raft_consensus.cc:3032] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:57.625176 32471 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260806 04:01:57.626092 1137 leader_election.cc:302] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.620515 1186 rpcz_store.cc:276] 0806 04:01:56.562992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.587851 (+1024859us) service_pool.cc:227] Handling call
0806 04:01:57.588430 (+ 579us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:57.607910 (+ 19480us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.608253 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:57.629078 1189 raft_consensus.cc:1226] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 5. Current term is 7. Ops: []
I20260806 04:01:57.571502 32739 raft_consensus.cc:1261] T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 04:01:57.627969 4993 raft_consensus.cc:2726] T 11b58a6c0e4e409fb675ad1e01ef8b60 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:57.632342 5481 raft_consensus.cc:2781] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:57.649150 1184 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260806 04:01:57.603611 5627 raft_consensus.cc:482] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:57.635869 1137 leader_election.cc:302] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.610452 5713 raft_consensus.cc:482] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:57.623288 5670 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:57.615657 5785 raft_consensus.cc:2726] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 04:01:57.647480 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2148) took 2926 ms. Trace:
W20260806 04:01:57.681214 32683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 279388us
I20260806 04:01:57.624300 5430 consensus_queue.cc:1035] T 4b2118a2ac1a4e3b873d360b7ef9b47e P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
W20260806 04:01:57.650740 32740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33182 (request call id 713) took 2169 ms (client timeout 2237 ms). Trace:
W20260806 04:01:57.686168 32740 rpcz_store.cc:269] 0806 04:01:55.480919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:55.481085 (+ 166us) service_pool.cc:227] Handling call
0806 04:01:57.650536 (+2169451us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54003a61d4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:57.650680 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3497451904}
W20260806 04:01:57.686795 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 51030.4us
I20260806 04:01:57.636548 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:57.659683 32682 leader_election.cc:302] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:57.662452 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 884) took 1219 ms. Trace:
I20260806 04:01:57.681416 32467 rpcz_store.cc:276] 0806 04:01:54.720849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:54.722027 (+ 1178us) service_pool.cc:227] Handling call
0806 04:01:54.743922 (+ 21895us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:54.761166 (+ 17244us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:54.761196 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:54.761633 (+ 437us) log.cc:842] Serialized 26 byte log entry
0806 04:01:55.746602 (+984969us) spinlock_profiling.cc:243] Waited 7.91 ms on lock 0x7b18004e3940. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd1731cb8b1 00007fd1731d8661 00007fd16a30e345 00007fd173145a39 00007fd16a32b8bb 00007fd16a329cbf 00007fd16a32be22 00007fd16ced55bb 00007fd16cedb063 00007fd16cedb411 00007fd16d2285cf
0806 04:01:56.377380 (+630778us) raft_consensus.cc:1631] Marking committed up to 4
0806 04:01:56.377423 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.377498 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.379617 (+1002119us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:57.647263 (+267646us) raft_consensus.cc:1671] finished
0806 04:01:57.647282 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.647418 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3492,"raft.queue_time_us":6078,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":46707,"spinlock_wait_cycles":50282624,"thread_start_us":630539,"threads_started":1,"wal-append.queue_time_us":1849820}
W20260806 04:01:57.633886 5787 log.cc:924] Time spent T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.071s user 0.000s sys 0.003s
I20260806 04:01:57.624868 5380 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 04:01:57.604418 5473 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:57.648905 5742 log.cc:924] Time spent T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 1.033s user 0.005s sys 0.004s
W20260806 04:01:57.641877 5790 log.cc:924] Time spent T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.097s user 0.003s sys 0.000s
W20260806 04:01:57.643792 5793 log.cc:924] Time spent T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.055s user 0.000s sys 0.002s
W20260806 04:01:57.646951 5810 log.cc:924] Time spent T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.057s user 0.000s sys 0.002s
W20260806 04:01:57.647128 5577 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260806 04:01:57.710182 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:57.561326 5682 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.014s sys 0.004s
I20260806 04:01:57.682498 32553 leader_election.cc:302] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.650754 32473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2165) took 1414 ms. Trace:
I20260806 04:01:57.724670 32473 rpcz_store.cc:276] 0806 04:01:56.236824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.248105 (+ 11281us) service_pool.cc:227] Handling call
0806 04:01:56.248456 (+ 351us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:56.248819 (+ 363us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.248853 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:56.321240 (+ 72387us) log.cc:842] Serialized 52 byte log entry
0806 04:01:56.943668 (+622428us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:56.992757 (+ 49089us) log.cc:842] Serialized 14 byte log entry
0806 04:01:56.993169 (+ 412us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.015886 (+ 22717us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.650504 (+634618us) raft_consensus.cc:1671] finished
0806 04:01:57.650526 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.650688 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22415,"prepare.queue_time_us":48910,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":42,"raft.queue_time_us":71521,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":28554,"thread_start_us":587950,"threads_started":1,"wal-append.queue_time_us":962923}
I20260806 04:01:57.689461 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.726866 32749 raft_consensus.cc:2379] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 8: Already voted for candidate f7c202b562ce482db283463d7f02e136 in this term.
I20260806 04:01:57.688678 1190 rpcz_store.cc:276] 0806 04:01:56.443248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.444332 (+ 1084us) service_pool.cc:227] Handling call
0806 04:01:56.444729 (+ 397us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:56.445061 (+ 332us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.445087 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:56.445517 (+ 430us) log.cc:842] Serialized 26 byte log entry
0806 04:01:56.957207 (+511690us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:56.957263 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:56.957436 (+ 173us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.662235 (+704799us) raft_consensus.cc:1671] finished
0806 04:01:57.662253 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.662395 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":511306,"threads_started":1,"wal-append.queue_time_us":790119}
I20260806 04:01:57.690575 32475 raft_consensus.cc:1226] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [term 5 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:57.689556 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725b0e6610ef466d8d7cf9ed945d2d4e" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.634871 5442 raft_consensus.cc:482] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:57.636778 5544 raft_consensus.cc:2726] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 04:01:57.634374 5805 raft_consensus.cc:2726] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:57.633250 5423 raft_consensus.cc:729] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:57.654341 5481 raft_consensus.cc:482] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:57.745801 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.698493 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.699168 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:57.638721 5809 consensus_queue.cc:1046] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260806 04:01:57.663795 5627 raft_consensus.cc:3037] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 11 FOLLOWER]: Advancing to term 12
I20260806 04:01:57.700991 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.670892 5713 raft_consensus.cc:3037] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:57.685710 5670 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:57.662352 5543 raft_consensus.cc:3037] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:57.708645 32743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.802309 32743 raft_consensus.cc:2379] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 9: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
I20260806 04:01:57.721553 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c8a38c9f554ef887e7782aa21ad891" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.735055 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:57.736709 32749 raft_consensus.cc:1261] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 04:01:57.699613 5252 raft_consensus.cc:2781] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:57.740420 32553 leader_election.cc:302] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:57.652581 5495 consensus_queue.cc:237] T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.734349 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:57.671123 4637 consensus_queue.cc:1046] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260806 04:01:57.642714 5612 raft_consensus.cc:504] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.694562 5380 raft_consensus.cc:482] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:57.830920 5805 raft_consensus.cc:3032] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:57.831377 5380 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:57.831470 5805 raft_consensus.cc:729] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:57.832417 5805 consensus_queue.cc:260] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.833797 5805 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:57.702107 4849 raft_consensus.cc:2781] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 04:01:57.722522 5682 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s user 0.014s sys 0.004s
I20260806 04:01:57.740346 5714 raft_consensus.cc:686] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 12 LEADER]: Becoming Leader. State: Replica: dd7344d85a154539b7f8c484f2a76772, State: Running, Role: LEADER
I20260806 04:01:57.837049 5714 consensus_queue.cc:237] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.831043 5612 leader_election.cc:288] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:57.757968 5473 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.758510 5825 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:57.703205 5470 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.766387 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 892) took 1217 ms. Trace:
I20260806 04:01:57.706182 4523 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:57.842796 5825 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.843156 4523 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.770196 1190 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 12 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 11. Current term is 12. Ops: []
I20260806 04:01:57.844987 1189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 1021) took 1030 ms. Trace:
I20260806 04:01:57.695566 32682 leader_election.cc:302] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541
I20260806 04:01:57.845202 5380 raft_consensus.cc:504] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.777262 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:57.808338 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 68011us
I20260806 04:01:57.846464 5610 raft_consensus.cc:2726] T 12e7971e809846ada28ebf1d6cea4837 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:57.846653 4523 leader_election.cc:288] T bd55a7d68ee84cb7a69c962f984f9372 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:57.741334 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9b09ac79f84fb797c2a263c612eee5" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:57.809670 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 890) took 1282 ms. Trace:
I20260806 04:01:57.745345 5674 raft_consensus.cc:482] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:57.813376 5110 raft_consensus.cc:482] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:57.813774 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 891) took 1285 ms. Trace:
I20260806 04:01:57.814386 1137 leader_election.cc:302] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.815320 32469 raft_consensus.cc:1261] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Refusing update from remote peer f7c202b562ce482db283463d7f02e136: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260806 04:01:57.816506 5820 log.cc:924] Time spent T 62779f9120ed4ef3b0d9febc7c2cd224 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20260806 04:01:57.641561 5337 consensus_queue.cc:237] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.845422 1189 rpcz_store.cc:276] 0806 04:01:56.814850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.637774 (+822924us) service_pool.cc:227] Handling call
0806 04:01:57.638158 (+ 384us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:57.638566 (+ 408us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:57.638594 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:57.639184 (+ 590us) log.cc:842] Serialized 52 byte log entry
0806 04:01:57.645922 (+ 6738us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:57.668792 (+ 22870us) log.cc:842] Serialized 14 byte log entry
0806 04:01:57.669028 (+ 236us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.669098 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.844768 (+175670us) raft_consensus.cc:1671] finished
0806 04:01:57.844791 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.844928 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23144,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":135,"raft.queue_time_us":2438,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":17394432,"thread_start_us":6278,"threads_started":1,"wal-append.queue_time_us":7877}
I20260806 04:01:57.803551 32739 raft_consensus.cc:2454] T 3c9b09ac79f84fb797c2a263c612eee5 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 9.
I20260806 04:01:57.759792 5827 raft_consensus.cc:2726] T 7ec5d99c8efa405b84db49741c2f8933 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 04:01:57.851369 32466 raft_consensus.cc:2379] T 3c9b09ac79f84fb797c2a263c612eee5 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 9: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:01:57.845855 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 153931us
I20260806 04:01:57.847299 1188 rpcz_store.cc:276] 0806 04:01:56.527662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.569939 (+ 42277us) service_pool.cc:227] Handling call
0806 04:01:56.601609 (+ 31670us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 04:01:56.602018 (+ 409us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.602049 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 04:01:57.272351 (+670302us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b2c00127298. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16bc37e11 00007fd16bc47442 00007fd16d28fc11 00007fd16d25458a 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772
0806 04:01:57.272892 (+ 541us) log.cc:842] Serialized 52 byte log entry
0806 04:01:57.538294 (+265402us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:57.538449 (+ 155us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.538564 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.765068 (+226504us) raft_consensus.cc:1671] finished
0806 04:01:57.765092 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.809596 (+ 44504us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7351,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":3146,"spinlock_wait_cycles":14772992,"thread_start_us":264397,"threads_started":1,"wal-append.queue_time_us":291546}
I20260806 04:01:57.750186 5481 raft_consensus.cc:3037] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:57.847736 1187 rpcz_store.cc:276] 0806 04:01:56.548916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.385152 (+836236us) service_pool.cc:227] Handling call
0806 04:01:57.385525 (+ 373us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:57.391742 (+ 6217us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:57.391776 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:57.401382 (+ 9606us) log.cc:842] Serialized 26 byte log entry
0806 04:01:57.536393 (+135011us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:57.536437 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.536513 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.766168 (+229655us) raft_consensus.cc:1671] finished
0806 04:01:57.766188 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.766317 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5820,"raft.queue_time_us":6403,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":14523,"thread_start_us":94192,"threads_started":1,"wal-append.queue_time_us":226769}
I20260806 04:01:57.854753 32427 leader_election.cc:302] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:57.857086 32467 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
W20260806 04:01:57.855590 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 231701us
I20260806 04:01:57.847266 5252 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:57.810210 5463 raft_consensus.cc:482] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:57.804987 5579 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:57.854060 1183 rpcz_store.cc:276] 0806 04:01:56.528754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.861439 (+332685us) service_pool.cc:227] Handling call
0806 04:01:56.883250 (+ 21811us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:56.885818 (+ 2568us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.885847 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:56.886274 (+ 427us) log.cc:842] Serialized 26 byte log entry
0806 04:01:57.236413 (+350139us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:57.242061 (+ 5648us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.242134 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:57.813564 (+571430us) raft_consensus.cc:1671] finished
0806 04:01:57.813586 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:57.813717 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":282,"raft.run_cpu_time_us":61,"raft.run_wall_time_us":62,"thread_start_us":345359,"threads_started":1,"wal-append.queue_time_us":701952}
W20260806 04:01:57.869338 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 55019.3us
I20260806 04:01:57.744386 5442 raft_consensus.cc:504] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:57.886013 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 366043us
I20260806 04:01:57.890364 32739 raft_consensus.cc:1226] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:57.886530 32605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2245) took 2053 ms. Trace:
I20260806 04:01:57.854082 4849 raft_consensus.cc:686] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Becoming Leader. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:57.907886 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:57.858175 4523 raft_consensus.cc:2781] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:57.933247 4523 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:57.933717 4523 raft_consensus.cc:3037] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:57.845016 5825 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:57.930220 32605 rpcz_store.cc:276] 0806 04:01:55.832608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:55.901663 (+ 69055us) service_pool.cc:227] Handling call
0806 04:01:56.500366 (+598703us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:56.500928 (+ 562us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.886347 (+1385419us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b5400334fd0. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:57.886475 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1326623488}
I20260806 04:01:57.884495 5473 raft_consensus.cc:2726] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260806 04:01:57.937991 1137 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.938236 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:57.938541 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec440a6dfdfc46af8a0876b464ef531d" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:57.939420 5473 raft_consensus.cc:2726] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:57.940537 32466 pending_rounds.cc:76] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Aborting all ops after (but not including) 6
I20260806 04:01:57.940842 32466 pending_rounds.cc:97] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 04:01:57.941073 32466 raft_consensus.cc:2866] T d20f5ce3f2ac40379306e3d630f63eda P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 04:01:57.942663 32602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:57.944181 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f39a49c08914cb0a772f3187e29fe61" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:57.944453 4523 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:57.945117 32609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c662c1a70954272bb9eff4a51caaeee" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:57.945490 1187 raft_consensus.cc:1261] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 11 LEARNER]: Refusing update from remote peer dd7344d85a154539b7f8c484f2a76772: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260806 04:01:57.945847 1137 leader_election.cc:302] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:57.946069 32605 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:57.946800 4523 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:57.947705 5470 raft_consensus.cc:2781] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:57.948110 5470 raft_consensus.cc:482] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting leader election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:57.948487 5470 raft_consensus.cc:3037] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:57.900449 5252 raft_consensus.cc:3037] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:57.857573 5674 raft_consensus.cc:504] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:57.947288 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:57.958679 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 104013us
I20260806 04:01:57.961527 32427 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541; no voters: f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.962947 1136 leader_election.cc:302] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:57.866984 5110 raft_consensus.cc:504] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:57.964670 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd55a7d68ee84cb7a69c962f984f9372" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:57.957363 5473 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:57.975483 32427 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772
W20260806 04:01:57.983317 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 136421us
I20260806 04:01:57.965068 5603 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:57.957932 5785 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:57.965176 5430 raft_consensus.cc:2726] T 6023572bff444c85842d7b3351debf87 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 04:01:57.910440 5463 raft_consensus.cc:504] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:58.016754 5834 log.cc:924] Time spent T 6023572bff444c85842d7b3351debf87 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260806 04:01:58.018011 5463 leader_election.cc:288] T e78ed11baa324b6bbbf7a965a2bad24e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:58.018478 5841 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260806 04:01:58.002898 32741 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:58.003983 1189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0bf6a30af60042f9b0a84eb0fdb8db19"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:58.020287 1189 raft_consensus.cc:595] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:57.997509 32609 raft_consensus.cc:2454] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 7.
I20260806 04:01:58.020653 1189 raft_consensus.cc:603] T 0bf6a30af60042f9b0a84eb0fdb8db19 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:57.924212 5579 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:57.983901 5473 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.014741 5840 raft_consensus.cc:2781] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 04:01:58.015429 5380 leader_election.cc:288] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:58.015759 5837 log.cc:924] Time spent T 0201a31e3c594fdfa1cec8a9970becef P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20260806 04:01:58.043123 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:58.016310 4849 consensus_queue.cc:237] T d2ad471021a947e9a45d57c7be704453 P d4a238f055a44f459c62c224daa4d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.007253 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:58.052757 32739 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:57.951948 5830 consensus_queue.cc:1046] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20260806 04:01:57.985363 5682 raft_consensus.cc:2726] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260806 04:01:58.032541 5836 log.cc:924] Time spent T c2611a58e8424bd088918c8aa63add87 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260806 04:01:57.995997 5488 raft_consensus.cc:2726] T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 04:01:58.000713 5495 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:58.057052 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 94187.5us
I20260806 04:01:58.065433 1136 leader_election.cc:302] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4a238f055a44f459c62c224daa4d541; no voters: dd7344d85a154539b7f8c484f2a76772, f7c202b562ce482db283463d7f02e136
I20260806 04:01:58.066720 32739 raft_consensus.cc:2454] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 6.
I20260806 04:01:58.032423 5110 leader_election.cc:288] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:58.033386 5840 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 04:01:58.069257 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 117641us
I20260806 04:01:58.073215 32428 leader_election.cc:302] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:58.067344 32682 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 88089.7us
I20260806 04:01:58.076476 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6ca7a29c2e4741a8285425bb994584" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:58.021967 5603 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.000530 5845 consensus_queue.cc:1035] T ab1948a6fbe64767a22598e910340a6b P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260806 04:01:58.043660 4679 consensus_queue.cc:1046] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260806 04:01:58.118927 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:58.129258 32739 raft_consensus.cc:1261] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260806 04:01:58.052157 4637 raft_consensus.cc:482] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:01:58.118923 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 131715us
I20260806 04:01:58.020578 5785 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.131006 32739 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03"
dest_uuid: "f7c202b562ce482db283463d7f02e136"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:38584
I20260806 04:01:58.133746 32605 raft_consensus.cc:2454] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 9.
I20260806 04:01:58.073452 5670 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:58.134008 32739 raft_consensus.cc:595] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:58.134027 32553 leader_election.cc:302] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:58.134706 32739 raft_consensus.cc:603] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:58.079247 5610 raft_consensus.cc:3032] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:58.079164 5840 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Advancing to term 10
W20260806 04:01:58.198282 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 67557.6us
I20260806 04:01:58.202590 32477 raft_consensus.cc:1226] T e4533f141e644c719df87540a1ca54e8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 4. Current term is 5. Ops: []
I20260806 04:01:58.116418 5682 raft_consensus.cc:2781] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 04:01:58.206979 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 73043.8us
I20260806 04:01:58.207805 32553 leader_election.cc:302] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:58.209156 32553 leader_election.cc:302] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd7344d85a154539b7f8c484f2a76772; no voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136
I20260806 04:01:58.212687 1190 raft_consensus.cc:1226] T 3f1e254fda3f4e30802dcd0bba7a82ca P d4a238f055a44f459c62c224daa4d541 [term 7 LEADER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:58.071530 5495 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.123402 5488 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:58.218209 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:58.224789 32405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 61718.8us
I20260806 04:01:58.225422 5430 consensus_queue.cc:1046] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 04:01:58.227159 5785 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.231248 5866 raft_consensus.cc:482] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:58.231887 5866 raft_consensus.cc:504] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.234154 5866 leader_election.cc:288] T 7ec5d99c8efa405b84db49741c2f8933 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:58.234761 5847 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.235344 5847 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.238193 5847 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.238804 5844 consensus_queue.cc:1035] T 4194210f40654ad28af7fc5644e71ace P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260806 04:01:58.240332 5697 raft_consensus.cc:2726] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:58.241297 5838 consensus_queue.cc:1046] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.190s
I20260806 04:01:58.241796 32739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:58.198442 5610 raft_consensus.cc:729] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:58.212525 5442 leader_election.cc:288] T c4f865676f884755987b8bd820489186 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:58.223580 5758 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260806 04:01:58.215133 5714 raft_consensus.cc:2726] T 725b0e6610ef466d8d7cf9ed945d2d4e P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:01:58.259626 5855 raft_consensus.cc:3037] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:58.213434 5606 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:58.217598 5682 raft_consensus.cc:686] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Leader. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
W20260806 04:01:58.255245 5851 log.cc:924] Time spent T 128baf4b99ba44f0aa14ba803db9e1ba P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
W20260806 04:01:58.256135 5852 log.cc:924] Time spent T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260806 04:01:58.221951 4993 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:58.216678 5579 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:58.198273 4637 raft_consensus.cc:504] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.286844 4637 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.250213 5843 consensus_queue.cc:1046] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.276s
I20260806 04:01:58.246871 5674 leader_election.cc:288] T 5c662c1a70954272bb9eff4a51caaeee P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.286742 5273 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Advancing to term 11
I20260806 04:01:58.270804 5032 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.266710 5670 raft_consensus.cc:482] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:58.298626 5603 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:58.298883 5670 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.299840 5473 leader_election.cc:288] T 187160a8bf734a23b90fcc49bc24b10b P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:58.299521 5682 consensus_queue.cc:237] T fe6ca7a29c2e4741a8285425bb994584 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.296556 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:58.304971 1184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 1022) took 1402 ms. Trace:
I20260806 04:01:58.297639 5032 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.299702 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:58.315054 32471 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
I20260806 04:01:58.278890 5873 consensus_queue.cc:1035] T e78ed11baa324b6bbbf7a965a2bad24e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20260806 04:01:58.243676 5488 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.305361 1184 rpcz_store.cc:276] 0806 04:01:56.902296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.663599 (+761303us) service_pool.cc:227] Handling call
0806 04:01:57.799979 (+136380us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:57.840341 (+ 40362us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:57.840375 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:57.840778 (+ 403us) log.cc:842] Serialized 26 byte log entry
0806 04:01:57.932704 (+ 91926us) raft_consensus.cc:1631] Marking committed up to 8
0806 04:01:57.934164 (+ 1460us) log.cc:842] Serialized 14 byte log entry
0806 04:01:57.934962 (+ 798us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.935230 (+ 268us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:58.304736 (+369506us) raft_consensus.cc:1671] finished
0806 04:01:58.304755 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:58.304909 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40074,"prepare.queue_time_us":897,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":41019,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":161,"thread_start_us":91592,"threads_started":1,"wal-append.queue_time_us":212005}
I20260806 04:01:58.295576 5610 consensus_queue.cc:260] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.314237 5032 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.322100 1184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:58.329547 1184 raft_consensus.cc:595] T 46d78e8748804374b8af237c1ca8ede8 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:58.337049 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:58.339205 1184 raft_consensus.cc:1226] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 9. Current term is 10. Ops: []
I20260806 04:01:58.345460 4637 raft_consensus.cc:482] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:58.395578 32683 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
W20260806 04:01:58.396929 5543 consensus_meta.cc:287] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.560s user 0.003s sys 0.008s
I20260806 04:01:58.397684 5543 consensus_queue.cc:1035] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260806 04:01:58.397950 5596 consensus_queue.cc:1046] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 04:01:58.399351 5596 raft_consensus.cc:482] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:58.387414 5221 consensus_meta.cc:287] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.788s user 0.010s sys 0.001s
I20260806 04:01:58.400216 5840 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.400631 5577 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:58.301297 4993 raft_consensus.cc:504] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.402112 32739 raft_consensus.cc:2396] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260806 04:01:58.374107 5032 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:58.409766 1184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:58.410530 1184 raft_consensus.cc:595] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Received request to transfer leadership to TS ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:58.410957 1184 raft_consensus.cc:603] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 04:01:58.416299 32738 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:58.430961 5422 consensus_queue.cc:1046] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 04:01:58.400627 5488 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:58.410279 5577 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.436913 5577 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.410283 5830 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:58.437731 5830 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:58.438073 5830 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:58.439374 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:58.441039 32466 raft_consensus.cc:2454] T 42a553c7b71d493284e5e79b6f19f848 P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 4.
I20260806 04:01:58.442509 32477 raft_consensus.cc:1226] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260806 04:01:58.393395 4922 raft_consensus.cc:3037] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:58.439540 5032 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.406349 5221 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.446343 5221 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:58.448836 5789 log.cc:924] Time spent T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.946s user 0.003s sys 0.000s
W20260806 04:01:58.410746 5341 consensus_meta.cc:287] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.819s user 0.011s sys 0.003s
I20260806 04:01:58.399818 5596 raft_consensus.cc:504] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.453308 5670 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.410200 5543 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.454700 5596 leader_election.cc:288] T 9f7858566354435c9b6b09c5b8c1714c P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:58.454382 5543 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.456946 5543 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.401099 4637 raft_consensus.cc:504] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.459000 4637 leader_election.cc:288] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.460459 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 898) took 1014 ms. Trace:
I20260806 04:01:58.460935 1185 rpcz_store.cc:276] 0806 04:01:57.445698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.463766 (+ 18068us) service_pool.cc:227] Handling call
0806 04:01:58.460197 (+996431us) spinlock_profiling.cc:243] Waited 55.8 ms on lock 0x7b540078e454. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:58.460376 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128234752}
I20260806 04:01:58.461995 4993 leader_election.cc:288] T e4533f141e644c719df87540a1ca54e8 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:58.462164 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a553c7b71d493284e5e79b6f19f848" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:58.437258 5495 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:58.464586 32604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42618 (request call id 793) took 1658 ms. Trace:
I20260806 04:01:58.464962 32604 rpcz_store.cc:276] 0806 04:01:56.806614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.807076 (+ 462us) service_pool.cc:227] Handling call
0806 04:01:56.818227 (+ 11151us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 04:01:56.818788 (+ 561us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.818817 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 04:01:56.819885 (+ 1068us) log.cc:842] Serialized 624 byte log entry
0806 04:01:57.209821 (+389936us) raft_consensus.cc:1631] Marking committed up to 5
0806 04:01:57.368229 (+158408us) log.cc:842] Serialized 14 byte log entry
0806 04:01:57.636125 (+267896us) log.cc:842] Serialized 14 byte log entry
0806 04:01:57.636397 (+ 272us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.636512 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:58.447969 (+811457us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:01:58.463361 (+ 15392us) raft_consensus.cc:1671] finished
0806 04:01:58.463388 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:58.464524 (+ 1136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17151,"prepare.queue_time_us":1054,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":68359,"raft.run_cpu_time_us":470,"raft.run_wall_time_us":13212,"spinlock_wait_cycles":53797760,"thread_start_us":612810,"threads_started":2,"wal-append.queue_time_us":602426}
I20260806 04:01:58.468015 32682 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:58.469168 5633 raft_consensus.cc:3032] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 04:01:58.481365 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:01:58.484280 5627 consensus_meta.cc:287] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.674s user 0.004s sys 0.008s
I20260806 04:01:58.484743 5627 raft_consensus.cc:504] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.485121 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:58.485719 32471 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 11.
W20260806 04:01:58.485785 5481 consensus_meta.cc:287] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.569s user 0.011s sys 0.000s
I20260806 04:01:58.487919 32683 leader_election.cc:302] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters: d4a238f055a44f459c62c224daa4d541
W20260806 04:01:58.493065 5252 consensus_meta.cc:287] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.531s user 0.004s sys 0.016s
I20260806 04:01:58.493583 5252 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.495795 5252 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.496670 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:58.500007 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
W20260806 04:01:58.505898 5713 consensus_meta.cc:287] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.658s user 0.009s sys 0.002s
I20260806 04:01:58.509218 1184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:58.512374 5713 raft_consensus.cc:504] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.512986 5855 raft_consensus.cc:2726] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 04:01:58.482405 5633 raft_consensus.cc:729] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:58.514140 5737 raft_consensus.cc:3032] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 04:01:58.514537 5737 raft_consensus.cc:729] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:58.515165 5713 leader_election.cc:288] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.473892 5519 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 04:01:58.514706 5633 consensus_queue.cc:260] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.508606 1185 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 11 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 11.
I20260806 04:01:58.522554 32682 leader_election.cc:302] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:58.505820 5341 raft_consensus.cc:2781] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 04:01:58.515868 5519 raft_consensus.cc:482] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:58.560473 5737 consensus_queue.cc:260] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.561872 5840 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Requested vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.562052 5737 raft_consensus.cc:3037] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:58.564254 5596 consensus_queue.cc:1046] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260806 04:01:58.564677 5519 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:58.565208 5596 raft_consensus.cc:3032] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 04:01:58.565577 5596 raft_consensus.cc:729] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:01:58.566375 5596 consensus_queue.cc:260] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.564729 5830 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.567549 5596 raft_consensus.cc:3037] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Advancing to term 10
I20260806 04:01:58.569540 5830 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941)
I20260806 04:01:58.571904 32738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:58.573387 5627 leader_election.cc:288] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 12 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.573733 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2081) took 1729 ms. Trace:
I20260806 04:01:58.574120 32615 rpcz_store.cc:276] 0806 04:01:56.844779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.860920 (+ 16141us) service_pool.cc:227] Handling call
0806 04:01:56.861245 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:56.917676 (+ 56431us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:56.917714 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:56.918168 (+ 454us) log.cc:842] Serialized 26 byte log entry
0806 04:01:57.368861 (+450693us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:57.368908 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.368996 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:58.570175 (+1201179us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b24002c57d8. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a184cf4 00007fd16a182dfb 00007fd16a211559 00007fd16a2111e5 00007fd16a210740 00007fd16cf03578 00007fd16cf03cd8 00007fd16cedb6e8 00007fd16ced6aea 00007fd16ced5f36
0806 04:01:58.573410 (+ 3235us) raft_consensus.cc:1671] finished
0806 04:01:58.573433 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:58.573672 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":278,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":8864,"spinlock_wait_cycles":11905920,"thread_start_us":431577,"threads_started":1,"wal-append.queue_time_us":652313}
I20260806 04:01:58.575388 32477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:58.576485 32477 raft_consensus.cc:3037] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:58.577339 1185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4f865676f884755987b8bd820489186"
dest_uuid: "d4a238f055a44f459c62c224daa4d541"
mode: GRACEFUL
new_leader_uuid: "dd7344d85a154539b7f8c484f2a76772"
from {username='slave'} at 127.0.0.1:45672
I20260806 04:01:58.578135 1185 raft_consensus.cc:595] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Received request to transfer leadership to TS dd7344d85a154539b7f8c484f2a76772
I20260806 04:01:58.578485 1185 raft_consensus.cc:603] T c4f865676f884755987b8bd820489186 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 04:01:58.544552 5805 consensus_meta.cc:287] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.710s user 0.010s sys 0.003s
W20260806 04:01:58.564844 5872 log.cc:924] Time spent T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20260806 04:01:58.586192 1187 raft_consensus.cc:2454] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7344d85a154539b7f8c484f2a76772 in term 9.
I20260806 04:01:58.587349 30646 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 04:01:58.558241 5341 raft_consensus.cc:482] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:58.613328 5341 raft_consensus.cc:3037] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:01:58.593797 5633 raft_consensus.cc:3037] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:58.516976 5481 raft_consensus.cc:504] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.558928 5252 raft_consensus.cc:2781] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260806 04:01:58.619125 32477 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:58.622004 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:58.648936 32683 leader_election.cc:302] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:01:58.596994 5519 raft_consensus.cc:504] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.686997 32743 raft_consensus.cc:1226] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:58.626268 5252 raft_consensus.cc:482] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:01:58.689836 5252 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 11 FOLLOWER]: Advancing to term 12
I20260806 04:01:58.674005 1192 raft_consensus.cc:1203] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Deduplicated request from leader. Original: 4.6->[5.7-5.7] Dedup: 5.7->[]
I20260806 04:01:58.694160 1190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 900) took 1247 ms. Trace:
I20260806 04:01:58.694563 1190 rpcz_store.cc:276] 0806 04:01:57.446231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.844734 (+398503us) service_pool.cc:227] Handling call
0806 04:01:57.845103 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:58.213147 (+368044us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:58.671675 (+458528us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b5400492ad4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:58.694094 (+ 22419us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":844933888}
I20260806 04:01:58.699443 1192 raft_consensus.cc:1226] T d20f5ce3f2ac40379306e3d630f63eda P d4a238f055a44f459c62c224daa4d541 [term 6 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260806 04:01:58.699842 1182 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 959) took 4926 ms (client timeout 4123 ms). Trace:
W20260806 04:01:58.701153 1136 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.003s sys 0.007s
W20260806 04:01:58.701481 1136 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.003s sys 0.007s
W20260806 04:01:58.700279 1182 rpcz_store.cc:269] 0806 04:01:53.747549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:53.747650 (+ 101us) service_pool.cc:227] Handling call
0806 04:01:55.940515 (+2192865us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b5800005498. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a2f4851 00007fd16a2f4702 00007fd1731e4b09 00007fd172120730 00007fd1721e679b 00007fd1721e6821 00007fd1721961b0 00007fd172195a94 00007fd16b87dc04 00007fd16b87db92
0806 04:01:58.673677 (+2733162us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400611554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:58.673868 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2347534592}
I20260806 04:01:58.703420 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"
I20260806 04:01:58.702400 32475 raft_consensus.cc:1226] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 8. Current term is 9. Ops: []
I20260806 04:01:58.715904 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f44dd8bdcea4806ab5987c15e9bc5b5" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:58.716781 1190 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:01:58.699105 5830 raft_consensus.cc:2781] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 04:01:58.689795 5519 leader_election.cc:288] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.702611 5620 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:58.770114 1142 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.030s sys 0.025s
W20260806 04:01:58.784735 1142 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.571s user 0.030s sys 0.025s
W20260806 04:01:58.793905 32551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.015s sys 0.049s
W20260806 04:01:58.794277 32551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.811s user 0.015s sys 0.049s
W20260806 04:01:58.770602 5470 consensus_meta.cc:287] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541: Time spent flushing consensus metadata: real 0.810s user 0.011s sys 0.001s
I20260806 04:01:58.767575 5840 raft_consensus.cc:482] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:01:58.768256 5032 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:58.781534 5579 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.807173 32605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:58.811535 32604 raft_consensus.cc:1226] T 4b2118a2ac1a4e3b873d360b7ef9b47e P dd7344d85a154539b7f8c484f2a76772 [term 12 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 11. Current term is 12. Ops: [11.12-11.12]
I20260806 04:01:58.805907 32609 raft_consensus.cc:1261] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260806 04:01:58.809201 5430 raft_consensus.cc:482] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:58.817345 32605 raft_consensus.cc:2379] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 in current term 5: Already voted for candidate d4a238f055a44f459c62c224daa4d541 in this term.
I20260806 04:01:58.793519 5478 raft_consensus.cc:482] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.816195 32604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7971e809846ada28ebf1d6cea4837" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:58.809568 32602 raft_consensus.cc:1226] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 04:01:58.821710 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4194210f40654ad28af7fc5644e71ace" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:58.792204 5830 raft_consensus.cc:686] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [term 9 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
W20260806 04:01:58.804908 5606 consensus_meta.cc:287] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772: Time spent flushing consensus metadata: real 0.530s user 0.007s sys 0.003s
I20260806 04:01:58.829492 32614 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P dd7344d85a154539b7f8c484f2a76772 [term 12 LEADER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
W20260806 04:01:58.830004 1188 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s user 0.022s sys 0.022s
I20260806 04:01:58.831770 32615 raft_consensus.cc:1261] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Refusing update from remote peer ae0b3e43e7a64f789e51018fd5518d23: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260806 04:01:58.834489 1188 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.815s user 0.022s sys 0.022s
W20260806 04:01:58.808683 1183 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.010s sys 0.018s
W20260806 04:01:58.837033 1183 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.633s user 0.010s sys 0.018s
I20260806 04:01:58.800393 5855 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.801455 5713 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:58.801056 5469 raft_consensus.cc:482] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:58.837370 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:58.783595 5620 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.857908 32614 raft_consensus.cc:1226] T d7c0dfae3c464cd18cfd95401b49eed9 P dd7344d85a154539b7f8c484f2a76772 [term 10 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 9. Current term is 10. Ops: []
W20260806 04:01:58.827026 5511 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.209s user 0.008s sys 0.025s
I20260806 04:01:58.811163 5519 raft_consensus.cc:482] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.819307 4922 raft_consensus.cc:482] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.843178 5478 raft_consensus.cc:504] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.874501 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 1025) took 1243 ms. Trace:
I20260806 04:01:58.831101 5520 raft_consensus.cc:482] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:58.883095 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78ed11baa324b6bbbf7a965a2bad24e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:58.920277 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ef18327cc345d08d82dbc954eaa969" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:58.883829 5541 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.924999 5541 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.878115 4922 raft_consensus.cc:504] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.887315 5478 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:58.889111 5857 raft_consensus.cc:482] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:58.897552 5485 raft_consensus.cc:3032] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 11 LEADER]: Stepping down as leader of term 11
I20260806 04:01:58.822989 5579 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.878832 5519 raft_consensus.cc:504] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.905113 5610 raft_consensus.cc:3037] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:01:58.908102 32467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 954) took 1201 ms. Trace:
W20260806 04:01:58.909073 32558 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.014s sys 0.014s
I20260806 04:01:58.909968 4679 raft_consensus.cc:3032] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 04:01:58.958915 4679 raft_consensus.cc:729] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:58.960261 4679 consensus_queue.cc:260] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.961961 4679 raft_consensus.cc:3037] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Advancing to term 8
I20260806 04:01:58.941776 5485 raft_consensus.cc:729] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:01:58.845570 5606 raft_consensus.cc:2726] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 04:01:58.923364 1190 raft_consensus.cc:2454] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:58.962986 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:01:58.962622 5857 raft_consensus.cc:504] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.963310 5485 consensus_queue.cc:260] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:58.864610 5511 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.251s user 0.008s sys 0.026s
I20260806 04:01:58.877058 1183 rpcz_store.cc:276] 0806 04:01:57.631299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.929237 (+297938us) service_pool.cc:227] Handling call
0806 04:01:57.955199 (+ 25962us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:57.955524 (+ 325us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:57.955556 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:57.955928 (+ 372us) log.cc:842] Serialized 26 byte log entry
0806 04:01:58.069531 (+113603us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:58.839063 (+769532us) log.cc:842] Serialized 14 byte log entry
0806 04:01:58.874099 (+ 35036us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:58.874197 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:58.874248 (+ 51us) raft_consensus.cc:1671] finished
0806 04:01:58.874259 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:58.874429 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33107,"prepare.queue_time_us":136117,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":6233,"raft.queue_time_us":39407,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":11655,"thread_start_us":718292,"threads_started":2,"wal-append.queue_time_us":337517}
I20260806 04:01:58.966513 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1c97829f044cfe84bbe23b612c3b81" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:01:58.968271 32608 raft_consensus.cc:1226] T 23ef18327cc345d08d82dbc954eaa969 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 7. Current term is 8. Ops: []
I20260806 04:01:58.967830 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128baf4b99ba44f0aa14ba803db9e1ba" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:58.805584 5470 raft_consensus.cc:504] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.815680 5840 raft_consensus.cc:504] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.971937 5840 leader_election.cc:288] T 9befcec6c17046459b03fa9ce98de94e P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:58.972795 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
W20260806 04:01:58.956483 32558 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.747s user 0.014s sys 0.014s
I20260806 04:01:58.954049 5830 consensus_queue.cc:237] T fb31133082b14fb889780e53b990e965 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.974114 1183 raft_consensus.cc:2454] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 11 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 11.
I20260806 04:01:58.964450 5485 raft_consensus.cc:3037] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23 [term 11 FOLLOWER]: Advancing to term 12
I20260806 04:01:58.844342 5430 raft_consensus.cc:504] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.995584 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:58.971068 5470 leader_election.cc:288] T 3c9b09ac79f84fb797c2a263c612eee5 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:58.956233 32467 rpcz_store.cc:276] 0806 04:01:57.706732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.706867 (+ 135us) service_pool.cc:227] Handling call
0806 04:01:57.737893 (+ 31026us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:57.882676 (+144783us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:58.907804 (+1025128us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400049cd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:58.908012 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273715456}
I20260806 04:01:58.963994 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d78e8748804374b8af237c1ca8ede8" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:58.962080 4922 leader_election.cc:288] T 77befd7e7cea42ed8a469471b80a08f7 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:58.981760 32738 raft_consensus.cc:2396] T 3c1c97829f044cfe84bbe23b612c3b81 P f7c202b562ce482db283463d7f02e136 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae0b3e43e7a64f789e51018fd5518d23 for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260806 04:01:58.982434 5519 leader_election.cc:288] T 3f1e254fda3f4e30802dcd0bba7a82ca P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:59.003610 32682 leader_election.cc:302] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136; no voters: ae0b3e43e7a64f789e51018fd5518d23
I20260806 04:01:58.860530 5855 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.876505 5469 raft_consensus.cc:504] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.982930 5481 leader_election.cc:288] T db65aabb33f8427593eac56e96ca9cc6 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 election: Requested vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:58.874684 5713 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.876127 5341 raft_consensus.cc:504] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:58.936362 5520 raft_consensus.cc:504] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:59.030362 5833 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.984333 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136"
I20260806 04:01:59.031154 5833 raft_consensus.cc:504] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:58.983045 32737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:59.032814 32737 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:01:58.986744 5579 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:01:58.992354 1187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37314 (request call id 1035) took 1045 ms. Trace:
I20260806 04:01:59.035935 1187 rpcz_store.cc:276] 0806 04:01:57.946941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.947073 (+ 132us) service_pool.cc:227] Handling call
0806 04:01:58.992187 (+1045114us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b5400710554. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:58.992292 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1434573568}
I20260806 04:01:59.038425 1187 raft_consensus.cc:1226] T f9fdfcc3345c4c86a7d974030c67c271 P d4a238f055a44f459c62c224daa4d541 [term 12 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 11. Current term is 12. Ops: [8.9-11.11]
I20260806 04:01:58.924206 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:58.965600 5857 leader_election.cc:288] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:58.997830 5273 raft_consensus.cc:482] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:59.006152 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:58.927738 5541 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:59.017591 1189 raft_consensus.cc:1226] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Rejecting Update request from peer ae0b3e43e7a64f789e51018fd5518d23 for earlier term 6. Current term is 7. Ops: []
I20260806 04:01:59.018671 1192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 1018) took 2832 ms. Trace:
W20260806 04:01:59.055418 1137 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.935s user 0.003s sys 0.031s
I20260806 04:01:59.055687 1192 rpcz_store.cc:276] 0806 04:01:56.186537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:56.202467 (+ 15930us) service_pool.cc:227] Handling call
0806 04:01:58.673858 (+2471391us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:58.704181 (+ 30323us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:59.018474 (+314293us) spinlock_profiling.cc:243] Waited 2.43 s on lock 0x7b5400611550. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d253a09 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:01:59.018608 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5595673344}
I20260806 04:01:59.019146 5600 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:58.942727 32615 raft_consensus.cc:1226] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Rejecting Update request from peer f7c202b562ce482db283463d7f02e136 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260806 04:01:59.048319 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 65567.1us
W20260806 04:01:59.078966 32469 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.858s user 0.027s sys 0.020s
W20260806 04:01:59.079433 32469 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.859s user 0.027s sys 0.020s
W20260806 04:01:59.049319 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 61912.8us
I20260806 04:01:58.999058 32610 raft_consensus.cc:2454] T 4194210f40654ad28af7fc5644e71ace P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 9.
I20260806 04:01:59.082183 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:59.051903 1182 raft_consensus.cc:3032] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 04:01:59.084203 1182 raft_consensus.cc:729] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4a238f055a44f459c62c224daa4d541, State: Running, Role: LEADER
I20260806 04:01:59.053469 1183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:01:59.051052 32614 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 8.
W20260806 04:01:59.055778 1137 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.937s user 0.003s sys 0.031s
W20260806 04:01:59.086263 1137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.129:35941, user_credentials={real_user=slave}} blocked reactor thread for 967514us
I20260806 04:01:59.086622 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7858566354435c9b6b09c5b8c1714c" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:59.087814 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb31133082b14fb889780e53b990e965" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:01:59.080744 32551 leader_election.cc:302] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4a238f055a44f459c62c224daa4d541, dd7344d85a154539b7f8c484f2a76772; no voters:
I20260806 04:01:59.090768 32427 leader_election.cc:302] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:01:59.082892 32610 raft_consensus.cc:2454] T fb31133082b14fb889780e53b990e965 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:01:59.050468 5273 raft_consensus.cc:504] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:59.085114 1182 consensus_queue.cc:260] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:59.092398 32746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:59.250690 5878 raft_consensus.cc:482] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:59.154493 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187160a8bf734a23b90fcc49bc24b10b" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:01:59.148854 32469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55048 (request call id 963) took 1072 ms. Trace:
I20260806 04:02:00.016155 32746 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 5.
I20260806 04:01:59.102707 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:01:59.572232 32748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.296s user 0.034s sys 0.052s
W20260806 04:02:00.018842 32748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.742s user 0.036s sys 0.054s
I20260806 04:01:59.092955 32467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:02:00.020597 1183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37300 (request call id 928) took 1017 ms. Trace:
W20260806 04:01:59.576643 5633 consensus_meta.cc:287] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 0.950s user 0.017s sys 0.000s
I20260806 04:02:00.021703 32471 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:01:59.037559 5596 raft_consensus.cc:2781] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 04:01:59.308437 5884 raft_consensus.cc:482] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:01:59.198294 5541 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:02:00.022913 32471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59514 (request call id 2199) took 1060 ms. Trace:
I20260806 04:02:00.022769 5884 raft_consensus.cc:504] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.023422 32471 rpcz_store.cc:276] 0806 04:01:58.962445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:58.962688 (+ 243us) service_pool.cc:227] Handling call
0806 04:02:00.022754 (+1060066us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400202bd4. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d2582ae 00007fd172195c27 00007fd16b87dc04 00007fd16b87db92 00007fd16b87dac1 00007fd16b87da3c 00007fd16b87ccf2 00007fd16b67e94c 00007fd16b67dd86 00007fd16b67d72f
0806 04:02:00.022856 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2433092352}
W20260806 04:01:59.054643 5849 log.cc:924] Time spent T e78ed11baa324b6bbbf7a965a2bad24e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.720s user 0.003s sys 0.003s
I20260806 04:01:59.726526 5627 raft_consensus.cc:482] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:01:59.572542 1185 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.008s sys 0.023s
I20260806 04:02:00.025903 1186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37314 (request call id 1023) took 3123 ms. Trace:
I20260806 04:02:00.015473 5855 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.026611 1186 rpcz_store.cc:276] 0806 04:01:56.902621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.703022 (+800401us) service_pool.cc:227] Handling call
0806 04:01:57.703321 (+ 299us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:57.703639 (+ 318us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:57.703665 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:57.731515 (+ 27850us) log.cc:842] Serialized 26 byte log entry
0806 04:01:57.978456 (+246941us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:01:57.978498 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:57.978568 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:58.916611 (+938043us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:02:00.025609 (+1108998us) raft_consensus.cc:1671] finished
0806 04:02:00.025631 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:02:00.025845 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1413,"raft.queue_time_us":800,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":31570,"spinlock_wait_cycles":9808256,"thread_start_us":179798,"threads_started":1,"wal-append.queue_time_us":325146}
W20260806 04:01:59.142196 32427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 51518us
W20260806 04:02:00.028976 1185 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.342s user 0.008s sys 0.032s
I20260806 04:01:59.143079 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:01:59.092801 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f7c202b562ce482db283463d7f02e136" is_pre_election: true
I20260806 04:02:00.031828 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2102) took 1063 ms. Trace:
W20260806 04:01:59.167678 32551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.132:36181, user_credentials={real_user=slave}} blocked reactor thread for 87017.9us
W20260806 04:01:59.141779 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 61550.6us
I20260806 04:02:00.021883 1182 raft_consensus.cc:3037] T fe6ca7a29c2e4741a8285425bb994584 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Advancing to term 6
I20260806 04:01:59.132068 32743 raft_consensus.cc:1226] T fe6ca7a29c2e4741a8285425bb994584 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Rejecting Update request from peer d4a238f055a44f459c62c224daa4d541 for earlier term 5. Current term is 6. Ops: []
I20260806 04:02:00.025658 5627 raft_consensus.cc:504] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:01:59.411649 32741 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.168s user 0.028s sys 0.055s
I20260806 04:01:59.158456 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b9e678e6049be86bca4e981f90379" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
W20260806 04:02:00.038478 32741 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.795s user 0.028s sys 0.069s
I20260806 04:02:00.043038 1183 rpcz_store.cc:276] 0806 04:01:59.003278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:59.021465 (+ 18187us) service_pool.cc:227] Handling call
0806 04:02:00.020530 (+999065us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:59.219831 32475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59514 (request call id 2184) took 1174 ms. Trace:
I20260806 04:02:00.050462 32475 rpcz_store.cc:276] 0806 04:01:58.045944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:58.046109 (+ 165us) service_pool.cc:227] Handling call
0806 04:01:58.046391 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:58.714771 (+668380us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:59.219554 (+504783us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b540003d254. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:01:59.219763 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1508234624}
W20260806 04:01:59.572439 1184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.003s sys 0.031s
W20260806 04:02:00.051955 1184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.440s user 0.003s sys 0.035s
I20260806 04:01:59.253065 5430 leader_election.cc:288] T 0bc7e4a68014409c8199e1e003add00a P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.050230 32615 rpcz_store.cc:276] 0806 04:01:58.968896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:59.078279 (+109383us) service_pool.cc:227] Handling call
0806 04:02:00.031764 (+953485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:02:00.068238 32610 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2103) took 1099 ms. Trace:
I20260806 04:02:00.030247 32427 leader_election.cc:302] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:02:00.068698 32610 rpcz_store.cc:276] 0806 04:01:58.969230 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:59.091929 (+122699us) service_pool.cc:227] Handling call
0806 04:02:00.068157 (+976228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:01:59.381628 5341 leader_election.cc:288] T 42a553c7b71d493284e5e79b6f19f848 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:01:59.160326 32553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.951s user 0.009s sys 0.028s
W20260806 04:02:00.070973 32553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.861s user 0.013s sys 0.059s
W20260806 04:02:00.072027 32553 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 1.86276e+06us
I20260806 04:02:00.017504 32469 rpcz_store.cc:276] 0806 04:01:58.076206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:58.077555 (+ 1349us) service_pool.cc:227] Handling call
0806 04:01:58.123015 (+ 45460us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:58.126741 (+ 3726us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:58.126800 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:58.127563 (+ 763us) log.cc:842] Serialized 26 byte log entry
0806 04:01:58.220096 (+ 92533us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:01:59.079806 (+859710us) log.cc:842] Serialized 14 byte log entry
0806 04:01:59.148513 (+ 68707us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:59.148600 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:01:59.148645 (+ 45us) raft_consensus.cc:1671] finished
0806 04:01:59.148656 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:01:59.148791 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61601,"prepare.queue_time_us":38234,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":61798,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":31558,"thread_start_us":864909,"threads_started":2,"wal-append.queue_time_us":85968}
I20260806 04:01:59.115691 5782 raft_consensus.cc:482] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:01:59.768509 5881 log.cc:924] Time spent T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260806 04:01:59.114459 32473 raft_consensus.cc:2932] T d7c0dfae3c464cd18cfd95401b49eed9 P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d4a238f055a44f459c62c224daa4d541 (127.29.237.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } attrs { replace: true } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } } }
W20260806 04:01:59.686935 4849 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.394s user 0.015s sys 0.044s
I20260806 04:01:59.167353 5422 raft_consensus.cc:2726] T 187160a8bf734a23b90fcc49bc24b10b P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 04:01:59.689327 5252 consensus_meta.cc:287] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 0.999s user 0.006s sys 0.012s
W20260806 04:01:59.319416 5594 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.091s user 0.020s sys 0.034s
I20260806 04:01:59.291532 5520 leader_election.cc:288] T bec6dcc3e9784bbe83c0ee7452223cb4 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:01:59.877540 5110 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.361s user 0.024s sys 0.044s
W20260806 04:02:00.102638 5110 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.586s user 0.028s sys 0.044s
W20260806 04:02:00.100937 5594 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.879s user 0.028s sys 0.053s
I20260806 04:02:00.091259 32614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2105) took 1121 ms. Trace:
I20260806 04:02:00.098812 5782 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.105090 32614 rpcz_store.cc:276] 0806 04:01:58.969813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:59.102404 (+132591us) service_pool.cc:227] Handling call
0806 04:02:00.091178 (+988774us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 04:02:00.103331 5252 raft_consensus.cc:504] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } attrs { replace: true } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:01:59.094972 5600 raft_consensus.cc:504] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.106829 5627 leader_election.cc:288] T 12e7971e809846ada28ebf1d6cea4837 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.095362 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff861725fdbf4efda0de572d435f5d6d" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.116907 32615 raft_consensus.cc:2396] T ff861725fdbf4efda0de572d435f5d6d P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7c202b562ce482db283463d7f02e136 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260806 04:01:59.093341 5857 raft_consensus.cc:482] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:01:59.154776 5796 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:01:59.381976 5833 leader_election.cc:288] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:59.292794 5713 leader_election.cc:288] T 305cbd07ce124048b7ac2f99b2aa40db P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:01:59.220486 5606 raft_consensus.cc:2781] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:01:59.205708 5136 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:59.705159 5469 leader_election.cc:288] T ff861725fdbf4efda0de572d435f5d6d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:01:59.421779 5620 leader_election.cc:288] T e5a23c953ef64542931fb8ede0fcbb97 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.096786 5824 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:02:00.097184 5878 raft_consensus.cc:504] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:02:00.075881 5876 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.078s user 0.011s sys 0.031s
I20260806 04:02:00.108518 5600 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.023254 5541 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:01:59.253882 5463 raft_consensus.cc:482] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:02:00.217060 5463 raft_consensus.cc:504] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:02:00.218709 5833 consensus_queue.cc:1035] T 11b58a6c0e4e409fb675ad1e01ef8b60 P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 04:02:00.219694 5463 leader_election.cc:288] T db915af860484020942e9495eda28804 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:02:00.109882 5273 leader_election.cc:288] T d2ad471021a947e9a45d57c7be704453 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.110734 5782 leader_election.cc:288] T 8feef3e400bd4fdda571ce5ead436f21 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.222394 5782 raft_consensus.cc:482] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:02:00.223060 5782 raft_consensus.cc:504] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:02:00.111380 5818 raft_consensus.cc:482] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:02:00.113286 5717 raft_consensus.cc:482] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:01:59.352222 5470 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
W20260806 04:01:59.380684 5682 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.026s sys 0.017s
I20260806 04:01:59.844973 5466 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:02:00.120407 32428 leader_election.cc:302] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23, f7c202b562ce482db283463d7f02e136; no voters:
I20260806 04:02:00.127494 32615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41324 (request call id 2100) took 1159 ms. Trace:
I20260806 04:01:59.254544 5423 consensus_queue.cc:260] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:02:00.164443 5584 raft_consensus.cc:482] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:02:00.169339 32608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41308 (request call id 2266) took 1360 ms. Trace:
W20260806 04:02:00.090662 5797 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.369s user 0.035s sys 0.047s
W20260806 04:02:00.097671 4849 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.804s user 0.021s sys 0.049s
I20260806 04:01:59.168620 5843 consensus_queue.cc:1046] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 1.000s
I20260806 04:01:59.093101 5827 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 1.026s
I20260806 04:02:00.094878 5430 raft_consensus.cc:2726] T 46d78e8748804374b8af237c1ca8ede8 P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 04:02:00.193799 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db65aabb33f8427593eac56e96ca9cc6" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772"
I20260806 04:02:00.168239 5855 raft_consensus.cc:482] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:02:00.207659 1182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 1036) took 1224 ms. Trace:
I20260806 04:02:00.025923 5884 leader_election.cc:288] T 725b0e6610ef466d8d7cf9ed945d2d4e P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:02:00.210996 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec5d99c8efa405b84db49741c2f8933" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.173383 5520 raft_consensus.cc:482] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:02:00.205158 5136 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.208887 5110 raft_consensus.cc:482] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:02:00.216049 5878 leader_election.cc:288] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:02:00.216235 5541 leader_election.cc:288] T db915af860484020942e9495eda28804 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:02:00.243046 5797 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.198s user 0.037s sys 0.047s
I20260806 04:02:00.220273 32467 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P ae0b3e43e7a64f789e51018fd5518d23 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:02:00.108925 5252 leader_election.cc:288] T d24bcc94fb734c95849f4cb89c15ca8e P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 12 election: Requested vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:02:00.224411 32605 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.380s user 0.015s sys 0.019s
W20260806 04:02:00.259794 32605 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.415s user 0.015s sys 0.019s
I20260806 04:02:00.183934 5606 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:02:00.209573 4679 consensus_meta.cc:287] T 23ef18327cc345d08d82dbc954eaa969 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 1.247s user 0.008s sys 0.007s
I20260806 04:02:00.179107 5857 raft_consensus.cc:504] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.207301 5596 raft_consensus.cc:686] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [term 7 LEADER]: Becoming Leader. State: Replica: f7c202b562ce482db283463d7f02e136, State: Running, Role: LEADER
I20260806 04:02:00.234457 32615 rpcz_store.cc:276] 0806 04:01:58.968054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:02:00.092556 (+1124502us) service_pool.cc:227] Handling call
0806 04:02:00.127421 (+ 34865us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 04:02:00.116022 32559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.398s user 0.015s sys 0.065s
W20260806 04:02:00.268055 32559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.058s user 0.016s sys 0.065s
I20260806 04:02:00.209748 5469 raft_consensus.cc:482] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:02:00.208563 5548 raft_consensus.cc:482] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
W20260806 04:02:00.236769 32428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.131:34975, user_credentials={real_user=slave}} blocked reactor thread for 140399us
I20260806 04:02:00.274344 32428 leader_election.cc:302] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae0b3e43e7a64f789e51018fd5518d23; no voters: d4a238f055a44f459c62c224daa4d541, f7c202b562ce482db283463d7f02e136
I20260806 04:02:00.236338 32608 rpcz_store.cc:276] 0806 04:01:58.809160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:58.810629 (+ 1469us) service_pool.cc:227] Handling call
0806 04:01:58.845070 (+ 34441us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:01:58.968661 (+123591us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:02:00.169040 (+1200379us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b5400353754. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16d1d9471 00007fd16d254f51 00007fd16d25399e 00007fd1721952e9 00007fd16b87b684 00007fd16b87b612 00007fd16b87b541 00007fd16b87b4bc 00007fd16b87a772 00007fd16b67e94c 00007fd16b67dd86
0806 04:02:00.169267 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":282891520}
I20260806 04:02:00.182425 5796 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:02:00.222586 5485 consensus_meta.cc:287] T f9fdfcc3345c4c86a7d974030c67c271 P ae0b3e43e7a64f789e51018fd5518d23: Time spent flushing consensus metadata: real 1.228s user 0.008s sys 0.008s
I20260806 04:02:00.244341 1182 rpcz_store.cc:276] 0806 04:01:58.982867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:59.051378 (+ 68511us) service_pool.cc:227] Handling call
0806 04:01:59.051743 (+ 365us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 04:02:00.207290 (+1155547us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:02:00.207328 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 04:02:00.207371 (+ 43us) raft_consensus.cc:1631] Marking committed up to 7
0806 04:02:00.207403 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:02:00.207467 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:02:00.207598 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9725}
W20260806 04:02:00.236236 5682 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.767s user 0.038s sys 0.030s
I20260806 04:02:00.251261 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
W20260806 04:02:00.216463 5876 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.975s user 0.012s sys 0.032s
I20260806 04:02:00.237721 5470 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.296877 5470 leader_election.cc:288] T e870f9de1d6a4db4bbf149c6a785135c P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:02:00.244376 5584 raft_consensus.cc:504] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.298936 5584 leader_election.cc:288] T 3c1c97829f044cfe84bbe23b612c3b81 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:02:00.227155 32609 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.363s user 0.008s sys 0.026s
W20260806 04:02:00.299674 32609 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.436s user 0.008s sys 0.026s
I20260806 04:02:00.227730 5824 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.301977 5824 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.270751 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.303829 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9befcec6c17046459b03fa9ce98de94e" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.255762 5423 raft_consensus.cc:3037] T d80edf8f3dcd4b64a167796d0c741ecd P ae0b3e43e7a64f789e51018fd5518d23 [term 4 FOLLOWER]: Advancing to term 5
I20260806 04:02:00.273512 5600 raft_consensus.cc:482] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:02:00.282106 32614 raft_consensus.cc:2454] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
I20260806 04:02:00.263641 5430 raft_consensus.cc:2781] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 04:02:00.258562 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feef3e400bd4fdda571ce5ead436f21" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.283255 5627 raft_consensus.cc:482] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:02:00.283313 4849 raft_consensus.cc:482] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:02:00.290926 5825 raft_consensus.cc:482] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:02:00.289305 5136 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.252768 5520 raft_consensus.cc:504] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.518092 5627 raft_consensus.cc:504] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:02:00.518870 5600 raft_consensus.cc:504] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.523677 5520 leader_election.cc:288] T 57c0e905630e43668379f2f95684775f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.264808 5633 consensus_queue.cc:1046] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.221s
I20260806 04:02:00.480070 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc7e4a68014409c8199e1e003add00a" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.473732 5584 raft_consensus.cc:482] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:02:00.231801 5818 raft_consensus.cc:504] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.249522 5855 raft_consensus.cc:504] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.487394 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec6dcc3e9784bbe83c0ee7452223cb4" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.500535 32610 raft_consensus.cc:2379] T 8feef3e400bd4fdda571ce5ead436f21 P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4a238f055a44f459c62c224daa4d541 in current term 8: Already voted for candidate dd7344d85a154539b7f8c484f2a76772 in this term.
I20260806 04:02:00.254244 5110 raft_consensus.cc:504] T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
W20260806 04:02:00.302062 5894 log.cc:924] Time spent T d7c0dfae3c464cd18cfd95401b49eed9 P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20260806 04:02:00.501053 32608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "d4a238f055a44f459c62c224daa4d541" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.249742 5466 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P f7c202b562ce482db283463d7f02e136 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.265341 5606 raft_consensus.cc:3037] T 1e2f839ac9f94c2f980cc895f641607f P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:02:00.266811 5833 consensus_queue.cc:1035] T d24bcc94fb734c95849f4cb89c15ca8e P dd7344d85a154539b7f8c484f2a76772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.212s
I20260806 04:02:00.545087 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c271fbb1b29c41b8afbfd003a9c9cb3b" candidate_uuid: "ae0b3e43e7a64f789e51018fd5518d23" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.348421 5485 raft_consensus.cc:2726] T 3c1c97829f044cfe84bbe23b612c3b81 P ae0b3e43e7a64f789e51018fd5518d23 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 04:02:00.235704 5717 raft_consensus.cc:504] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.484553 5430 raft_consensus.cc:482] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:02:00.474227 5824 raft_consensus.cc:482] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae0b3e43e7a64f789e51018fd5518d23)
I20260806 04:02:00.579463 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77befd7e7cea42ed8a469471b80a08f7" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.520678 4679 raft_consensus.cc:482] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dd7344d85a154539b7f8c484f2a76772)
I20260806 04:02:00.524217 5843 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:02:00.584970 32614 raft_consensus.cc:2454] T c271fbb1b29c41b8afbfd003a9c9cb3b P dd7344d85a154539b7f8c484f2a76772 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0b3e43e7a64f789e51018fd5518d23 in term 8.
W20260806 04:02:00.657089 32604 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.815s user 0.016s sys 0.040s
I20260806 04:02:00.280997 5469 raft_consensus.cc:504] T 62779f9120ed4ef3b0d9febc7c2cd224 P d4a238f055a44f459c62c224daa4d541 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.616953 32610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e254fda3f4e30802dcd0bba7a82ca" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.657852 32610 raft_consensus.cc:2454] T 3f1e254fda3f4e30802dcd0bba7a82ca P dd7344d85a154539b7f8c484f2a76772 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 7.
I20260806 04:02:00.283963 5548 raft_consensus.cc:504] T bd55a7d68ee84cb7a69c962f984f9372 P ae0b3e43e7a64f789e51018fd5518d23 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.523945 5600 leader_election.cc:288] T 2f39a49c08914cb0a772f3187e29fe61 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.523958 5627 leader_election.cc:288] T ab1948a6fbe64767a22598e910340a6b P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:02:00.571616 32608 raft_consensus.cc:2454] T e4533f141e644c719df87540a1ca54e8 P dd7344d85a154539b7f8c484f2a76772 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4a238f055a44f459c62c224daa4d541 in term 4.
I20260806 04:02:00.527498 5541 raft_consensus.cc:482] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d4a238f055a44f459c62c224daa4d541)
I20260806 04:02:00.545207 5470 raft_consensus.cc:482] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
I20260806 04:02:00.571688 5337 raft_consensus.cc:482] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f7c202b562ce482db283463d7f02e136)
W20260806 04:02:00.650456 5546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.759s user 0.003s sys 0.066s
W20260806 04:02:00.630205 32602 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.774s user 0.023s sys 0.038s
I20260806 04:02:00.518316 4849 raft_consensus.cc:504] T 305cbd07ce124048b7ac2f99b2aa40db P d4a238f055a44f459c62c224daa4d541 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.618194 5884 raft_consensus.cc:482] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 04:02:00.653833 32612 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.718s user 0.023s sys 0.031s
I20260806 04:02:00.651024 32615 raft_consensus.cc:2454] T 77befd7e7cea42ed8a469471b80a08f7 P dd7344d85a154539b7f8c484f2a76772 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 9.
I20260806 04:02:00.658036 32614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30158f7dfb0455780b91c51e03665bc" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dd7344d85a154539b7f8c484f2a76772" is_pre_election: true
I20260806 04:02:00.670547 32614 raft_consensus.cc:2454] T e30158f7dfb0455780b91c51e03665bc P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
W20260806 04:02:00.671335 32612 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.735s user 0.023s sys 0.032s
W20260806 04:02:00.657511 32604 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.816s user 0.016s sys 0.040s
I20260806 04:02:00.577256 5584 raft_consensus.cc:504] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.521739 5825 raft_consensus.cc:504] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:02:00.677416 5584 leader_election.cc:288] T ec440a6dfdfc46af8a0876b464ef531d P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:02:00.637629 5782 leader_election.cc:288] T 8417acc05db341aca15704cd05736651 P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:02:00.583294 5596 consensus_queue.cc:237] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P f7c202b562ce482db283463d7f02e136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:02:00.700670 5908 log.cc:924] Time spent T 725b0e6610ef466d8d7cf9ed945d2d4e P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260806 04:02:00.701839 5855 leader_election.cc:288] T d20f5ce3f2ac40379306e3d630f63eda P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.702533 5857 leader_election.cc:288] T 128baf4b99ba44f0aa14ba803db9e1ba P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
W20260806 04:02:00.703080 5897 log.cc:924] Time spent T 4b2118a2ac1a4e3b873d360b7ef9b47e P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 1.065s user 0.017s sys 0.002s
I20260806 04:02:00.704694 1188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37290 (request call id 1026) took 3073 ms. Trace:
I20260806 04:02:00.705042 1188 rpcz_store.cc:276] 0806 04:01:57.631550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:57.923448 (+291898us) service_pool.cc:227] Handling call
0806 04:01:57.954682 (+ 31234us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:57.955029 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:57.955075 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:57.955463 (+ 388us) log.cc:842] Serialized 26 byte log entry
0806 04:01:58.836539 (+881076us) raft_consensus.cc:1631] Marking committed up to 10
0806 04:01:58.836587 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:01:58.836670 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:02:00.702512 (+1865842us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b44000232cc. stack: 00007fd16a2fa033 00007fd16a2f9f30 00007fd169c58534 00007fd1739648b4 00007fd173964849 00007fd16a170f11 00007fd16a23ea95 00007fd16a23f4b2 00007fd16cee7511 00007fd16cee2e1a 00007fd16ced6bee 00007fd16ced5f36 00007fd16ced5b58 00007fd16cedf551 00007fd16cedf509 00007fd16cedf499
0806 04:02:00.704353 (+ 1841us) raft_consensus.cc:1671] finished
0806 04:02:00.704378 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:02:00.704630 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":94,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":6394,"spinlock_wait_cycles":991671680,"thread_start_us":816762,"threads_started":1,"wal-append.queue_time_us":1462397}
W20260806 04:02:00.708475 5916 log.cc:924] Time spent T d7c0dfae3c464cd18cfd95401b49eed9 P d4a238f055a44f459c62c224daa4d541: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260806 04:02:00.709858 1185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37300 (request call id 913) took 2444 ms. Trace:
I20260806 04:02:00.710222 1185 rpcz_store.cc:276] 0806 04:01:58.265938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 04:01:58.579646 (+313708us) service_pool.cc:227] Handling call
0806 04:01:58.620243 (+ 40597us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 04:01:58.620592 (+ 349us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 04:01:58.620621 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 04:01:58.686140 (+ 65519us) log.cc:842] Serialized 26 byte log entry
0806 04:02:00.029475 (+1343335us) raft_consensus.cc:1631] Marking committed up to 6
0806 04:02:00.029520 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 04:02:00.029611 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 04:02:00.709658 (+680047us) raft_consensus.cc:1671] finished
0806 04:02:00.709678 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 04:02:00.709802 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1342860,"threads_started":1,"wal-append.queue_time_us":1833153}
I20260806 04:02:00.718555 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bcc94fb734c95849f4cb89c15ca8e" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541"
I20260806 04:02:00.719269 1182 raft_consensus.cc:3037] T d24bcc94fb734c95849f4cb89c15ca8e P d4a238f055a44f459c62c224daa4d541 [term 11 LEARNER]: Advancing to term 12
I20260806 04:02:00.719641 1187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45b7481674ca403b9a224047a6d99b03" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:02:00.719900 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ad471021a947e9a45d57c7be704453" candidate_uuid: "f7c202b562ce482db283463d7f02e136" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4a238f055a44f459c62c224daa4d541" is_pre_election: true
I20260806 04:02:00.720315 1187 raft_consensus.cc:2454] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c202b562ce482db283463d7f02e136 in term 5.
I20260806 04:02:00.656483 5430 raft_consensus.cc:3037] T c271fbb1b29c41b8afbfd003a9c9cb3b P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Advancing to term 9
I20260806 04:02:00.657146 5824 raft_consensus.cc:504] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } }
I20260806 04:02:00.677644 5825 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:02:00.667775 5546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.776s user 0.003s sys 0.066s
I20260806 04:02:00.659813 4679 raft_consensus.cc:504] T e870f9de1d6a4db4bbf149c6a785135c P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.662871 5843 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.669862 5470 raft_consensus.cc:504] T 65c8a38c9f554ef887e7782aa21ad891 P d4a238f055a44f459c62c224daa4d541 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.668495 5541 raft_consensus.cc:504] T 0201a31e3c594fdfa1cec8a9970becef P ae0b3e43e7a64f789e51018fd5518d23 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.670034 5337 raft_consensus.cc:504] T d7172fff185b4788acb609d2314e4b4b P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.675311 5884 raft_consensus.cc:504] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "dd7344d85a154539b7f8c484f2a76772" member_type: VOTER last_known_addr { host: "127.29.237.130" port: 37857 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
I20260806 04:02:00.727253 5796 leader_election.cc:288] T d80edf8f3dcd4b64a167796d0c741ecd P f7c202b562ce482db283463d7f02e136 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
W20260806 04:02:00.748346 5527 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.043s sys 0.104s
W20260806 04:02:00.750378 5527 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.982s user 0.043s sys 0.105s
W20260806 04:02:00.748785 5610 consensus_meta.cc:287] T 2f39a49c08914cb0a772f3187e29fe61 P f7c202b562ce482db283463d7f02e136: Time spent flushing consensus metadata: real 1.747s user 0.015s sys 0.000s
W20260806 04:02:00.750468 5896 log.cc:924] Time spent T d7172fff185b4788acb609d2314e4b4b P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.173s user 0.001s sys 0.001s
I20260806 04:02:00.753994 5824 leader_election.cc:288] T 18f3a6d6cef24470a69dbafa39b3d3e1 P dd7344d85a154539b7f8c484f2a76772 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:02:00.756795 5717 leader_election.cc:288] T a12b9e678e6049be86bca4e981f90379 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
W20260806 04:02:00.761777 5903 log.cc:924] Time spent T 4194210f40654ad28af7fc5644e71ace P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260806 04:02:00.756042 5818 leader_election.cc:288] T 45b7481674ca403b9a224047a6d99b03 P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), f7c202b562ce482db283463d7f02e136 (127.29.237.131:34975)
I20260806 04:02:00.775846 5843 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857), d4a238f055a44f459c62c224daa4d541 (127.29.237.132:36181)
I20260806 04:02:00.789229 5878 raft_consensus.cc:3032] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 04:02:00.789716 5878 raft_consensus.cc:729] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae0b3e43e7a64f789e51018fd5518d23, State: Running, Role: LEADER
I20260806 04:02:00.790560 5878 consensus_queue.cc:260] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ae0b3e43e7a64f789e51018fd5518d23" member_type: VOTER last_known_addr { host: "127.29.237.129" port: 35941 } } peers { permanent_uuid: "f7c202b562ce482db283463d7f02e136" member_type: VOTER last_known_addr { host: "127.29.237.131" port: 34975 } } peers { permanent_uuid: "d4a238f055a44f459c62c224daa4d541" member_type: VOTER last_known_addr { host: "127.29.237.132" port: 36181 } attrs { promote: false } }
W20260806 04:02:00.781879 5902 log.cc:924] Time spent T ab1948a6fbe64767a22598e910340a6b P f7c202b562ce482db283463d7f02e136: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20260806 04:02:00.778690 5884 leader_election.cc:288] T 1e2f839ac9f94c2f980cc895f641607f P d4a238f055a44f459c62c224daa4d541 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae0b3e43e7a64f789e51018fd5518d23 (127.29.237.129:35941), dd7344d85a154539b7f8c484f2a76772 (127.29.237.130:37857)
I20260806 04:02:00.798547 5878 raft_consensus.cc:3037] T 4f44dd8bdcea4806ab5987c15e9bc5b5 P ae0b3e43e7a64f789e51018fd5518d23 [term 6 FOLLOWER]: Advancing to term 7
I20260806 04:02:00.833420 30646 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 04:02:00.835819 30646 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 04:02:00.837128 32430 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.694s user 0.030s sys 0.053s
W20260806 04:02:00.847771 32430 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.705s user 0.031s sys 0.053s
W20260806 04:02:00.852634 1136 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.444s user 0.033s sys 0.043s
W20260806 04:02:00.853155 1136 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.445s user 0.033s sys 0.044s
W20260806 04:02:00.853883 1136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.237.130:37857, user_credentials={real_user=slave}} blocked reactor thread for 1.44604e+06us
I20260806 04:02:00.864287 32475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2f839ac9f94c2f980cc895f641607f" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
W20260806 04:02:00.865700 1136 connection.cc:232] RPC call timeout handler was delayed by -1.3684632338117808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
I20260806 04:02:00.866336 32475 raft_consensus.cc:2379] T 1e2f839ac9f94c2f980cc895f641607f P ae0b3e43e7a64f789e51018fd5518d23 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd7344d85a154539b7f8c484f2a76772 in current term 9: Already voted for candidate ae0b3e43e7a64f789e51018fd5518d23 in this term.
W20260806 04:02:00.837594 32326 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39782 (request call id 333) took 29342 ms (client timeout 15000 ms). Trace:
I20260806 04:02:00.839864 32609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2085) took 2026 ms. Trace:
W20260806 04:02:00.840307 32603 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.912s user 0.032s sys 0.037s
W20260806 04:02:00.867095 1136 connection.cc:232] RPC call timeout handler was delayed by -1.0706094768829644s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21640000000000004s for a response
I20260806 04:02:00.841533 32612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41324 (request call id 2092) took 2023 ms. Trace:
I20260806 04:02:00.875866 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4533f141e644c719df87540a1ca54e8" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23" is_pre_election: true
I20260806 04:02:00.842434 32741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33994 (request call id 2542) took 2600 ms. Trace:
I20260806 04:02:00.876327 32470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c0dfae3c464cd18cfd95401b49eed9" candidate_uuid: "dd7344d85a154539b7f8c484f2a76772" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae0b3e43e7a64f789e51018fd5518d23"